Digital Edge

The Microsoft Dynamics Slow Down

Michael Petrov — Tue, 29 Jan 2013 00:00:00

After purchasing a Microsoft Dynamics tool through a third party company, a client of ours encountered a severe speed problem. Typically a Microsoft SQL server will store best practice execution plans to a cache for queries. This way the plan can be instantly reused without any recalculation.

Pulling up the execution plan wasn’t so easy this time around. We ran the SQL profiler searching for answers and found that query was not actually defected but completely flooded! The execution plan had duplicated thousands of records of this same information. The query engine became so bloated with repeated files that it wasn’t able to pick up the correct plan, thereby causing a significant slowdown in the system’s performance.

We found that the server was not overloaded in terms of IO or CPU. When we tried to execute the query from SSMS it ran quickly.

Here is a sample of the query that was causing slowness when executed from Dynamics:

SELECT

A.JOURNALNUM,A.LINENUM,A.ACCOUNTTYPE,A.ACCOUNTNUM,A.COMPANY,A.TXT,

A.AMOUNTCURDEBIT,A.CURRENCYCODE,A.EXCHRATE,A.TAXGROUP,A.CASHDISCPERCENT,

A.QTY,A.DIMENSION,A.DIMENSION2_,A.DIMENSION3_,A.DIMENSION4_,A.DIMENSION5_,

A.DIMENSION6_,A.DIMENSION7_,A.DIMENSION8_,A.DIMENSION9_,A.DIMENSION10_,

A.BANKNEGINSTRECIPIENTNAME,A.ENTERPRISENUMBER,A.PRICE,A.INTERCODIMENSION,

A.INTERCODIMENSION2_,A.INTERCODIMENSION3_,A.INTERCODIMENSION4_,A.INTERCODIMENSION5_,

A.INTERCODIMENSION6_,A.INTERCODIMENSION7_,A.INTERCODIMENSION8_,A.INTERCODIMENSION9_,

A.INTERCODIMENSION10_,A.SETTLEVOUCHER,A.POSTINGPROFILE,A.VOUCHER,A.AMOUNTCURCREDIT,

A.PAYMENTSTATUS,A.CASHDISCAMOUNT,A.ACKNOWLEDGEMENTDATE,A.VENDTRANSID,A.CUSTTRANSID,A.REMAINAMOUNT,

A.TAXCODE,A.OFFSETACCOUNT,A.BANKDEPOSITVOUCHER,A.PAYMREFERENCE,A.BANKDEPOSITNUM,A.PAYMSPEC,

A.PAYMMODE,A.MARKEDINVOICE,A.OFFSETACCOUNTTYPE,A.BANKTRANSTYPE,A.OFFSETCOMPANY,A.OFFSETTXT,

A.TAX1099AMOUNT,A.TAX1099BOX,A.PAYMENTNOTES,A.MARKEDINVOICERECID,A.FOREIGNVOUCHER,

A.FOREIGNCOMPANY,A.TRANSFERRED,A.CANCEL,A.NOEDIT,A.INVISIBLE,A.TRANSDATE,A.TRANSACTIONTYPE,

A.DOCUMENTDATE,A.DOCUMENTNUM,A.APPROVED,A.APPROVEDBY,A.PAYMID,A.FREQVALUE,A.FREQCODE,A.DUE,

A.DATECASHDISC,A.TRANSFERREDBY,A.LOADINGDATE,A.FILECREATED,A.PAYMENTACCOUNT,A.INVOICE,

A.TRANSFERREDTO,A.TRANSFER,A.LASTTRANSFERRED,A.TRANSFERREDON,A.PURCHLEDGERPOSTING,A.BANKCHEQUENUM,A.PAYMENT,

A.CASHDISCCODE,A.EXCHRATESECOND,A.TRIANGULATION,A.TAXITEMGROUP,A.VATNUMJOURNAL,A.LISTCODE,A.FURTHERPOSTINGTYPE,

A.PURCHIDRANGE,A.POOLRECID,A.PREPAYMENT,A.IMPORTDATE,A.BANKCENTRALBANKPURPOSECODE,A.REASONREFRECID,A.BANKCENTRALBANKPURPOSETEXT,

A.CUSTVENDBANKACCOUNTID,A.PAYMENTSEQUENCENUM,A.ERRORCODEPAYMENT,A.FURTHERPOSTINGRECID,A.TAXDIRECTIONCONTROL,

A.CUSTVENDNEGINSTPROTESTREASON,A.BANKREMITTANCETYPE,A.BANKREMITTANCEFILEID,A.BANKACCOUNTID,

A.BANKPROMISSORYNOTENUM,A.PAYMINSTRUCTIONID1,A.PAYMINSTRUCTIONID2,A.PAYMINSTRUCTIONID3,

A.PAYMINSTRUCTIONID4,A.BANKBILLOFEXCHANGENUM,A.NEGINSTID,A.TAXWITHHOLDGROUP,A.TAX1099STATEAMOUNT,

A.TAX1099STATE,A.REMITTEECURRENCY,A.SALESTYPE_SA,A.SALESID_SA,A.SUBCONTRACTORNUM_SA,

A.CONTRACTNUM_SA,A.ASSETGROUP_SA,A.PROJID_SA,A.REVERSEENTRY,A.REVERSEDATE,A.REVRECID,A.INVOICEDECLARATIONID_IS,

A.BANKRECONCILEACCOUNTATPOST,A.BANKCHEQUEDEPOSITTRANSREFRECID,A.MARKEDINVOICECOMPANY,A.RELEASEDATE,

A.RELEASEDATETZID,A.CUSTEINVOICEPAYMDELIVERYNUM,A.CUSTEINVOICEPAYMSECTIONNUM,A.CUSTEINVOICEPAYMTRANSNUM,

A.GSTHSTTAXTYPE_CA,A.FININTERESTAMOUNTCUR,A.BORDEROSEQUENCENUM_BR,A.BORDEROPAYMREFERENCE_BR,A.ONEVOUCHER_BR,

A.FINEAMOUNTCUR,A.PAYMENTTYPE,A.PAYMENTWAY,A.COMPANYDOCUMENT,A.TAXWITHHOLDCALCULATE_TH,A.TAXWITHHOLDALTERNATEVENDOR8027,

A.TAXWITHHOLDITEMGROUP_TH,A.REFRECID,A.CREDITFLAG,A.TAXWITHHOLDEDITSTATUS,A.ATTACHMENTS_CN,A.VOUCHERTYPEID_CN,

A.LEDGERACCOUNT_CN,A.LEDGERACCOUNTOFFSET_CN,A.TRACE_CN,A.FINALUSER,A.CFOPID,A.SERVICEINVOICE,A.TAXWITHHOLDCALCULATE,

A.INVOICEFINECODE,A.INVOICEFININTERESTCODE,A.INVOICETYPE,A.CUSTSUFRAMA,A.CUSTSUFRAMAPISCOFINS,A.CUSTSUFRAMANUMBER,

A.VOUCHER_CN,A.TAXWITHHOLDFBTCATEGORYID_IN,A.TAXWITHHOLDFBTTRANSTYPE_IN,A.EXCISETARIFFCODE_IN,A.CUSTOMSTARIFFDIRECTION_IN,

A.CUSTOMSTARIFFCODE_IN,A.SALESTAXFORMTYPE_IN,A.ASSESSABLEVALUE_IN,A.MAXIMUMRETAILPRICE_IN,A.NAME_IN,A.ADDRESS_IN,

A.ECCNUMBER_IN,A.EXCISERECORDTYPE_IN,A.IECNUMBER_IN,A.STCNUMBER_IN,A.TIN_IN,A.SALESTAXREGISTRATIONNUMBER_IN,

A.SERVICECODE_IN,A.GTASERVICECATEGORY_IN,A.NONRECOVERABLEPERCENT_IN,A.VATGOODSTYPE_IN,A.STATE_IN,A.CONSIGNMENTNOTENUM_IN,

A.CUSTOMSDUTY_IN,A.TDSGROUP_IN,A.TCSGROUP_IN,A.NATUREOFASSESSEE_IN,A.TAN_IN,A.INTERCOMPANYCOMPANYID_IN,A.CITATAXBASE,

A.KUMREMITTANCEPRINTED,A.KUMEXTENDEDINVOICEID,A.MODIFIEDDATETIME,A.RECVERSION,A.RECID,B.REFRECID,B.ASSETID,B.BOOKID,B.TRANSTYPE,

B.COMPANY,B.BUDGETMODEL,B.BUDGETPOSTINGSTATUS,B.CONSUMPTIONQUANTITY,B.DEPRECIATIONSTARTDATE,B.DEPRECIATIONTIME,B.REFASSETID,

B.RESERVETRANSID,B.REVALUATIONAMOUNT,B.REVALUATIONTRANS,B.LOWVALUEPOOLTYPE_AU,B.LVPTRANSFERID_AU,B.ASSETGROUP_IN,B.RECVERSION,

B.RECID,C.NEGATIVENETBOOKVALUE,C.EXCEEDINGNETBOOKVALUE,C.RECID FROM LEDGERJOURNALTRANS A,LEDGERJOURNALTRANS_ASSET B,ASSETBOOK C

WHERE

((A.DATAAREAID=N'yus4') AND (A.JOURNALNUM=N'10013293'))

AND ((B.DATAAREAID=N'yus4') AND (A.RECID=B.REFRECID))

AND ((C.DATAAREAID=N'yus4') AND (((((C.ASSETID=A.ACCOUNTNUM) AND (C.BOOKID=B.BOOKID)) AND (A.ACCOUNTTYPE=5)) AND ((B.TRANSTYPE<>8) AND (B.TRANSTYPE<>9))) AND ((C.NEGATIVENETBOOKVALUE=0) OR (C.EXCEEDINGNETBOOKVALUE=0))))

ORDER BY A.DATAAREAID,A.ACCOUNTNUM,B.DATAAREAID,B.BOOKID

We have since cleared the execution plan cache.

DBCC FREEPROCCACHE

And we have also saw that token cache size was somewhat high, so cleared that also.

DBCC FREESYSTEMCACHE ('TokenAndPermUserStore')

Performance seemed to go back to normal after this. Although our research and troubleshooting led us to a quick fix solution, this solution may only be temporary. We are looking to unveil a permanent solution this problem.

VMWare Migration Gone Wrong (continued)

Michael Petrov — Mon, 07 Jan 2013 00:00:00

With a lot of troubleshooting and a bit of red tape, our team reached a solution to the trouble we faced during the migration process earlier. On the contrary of what was expected, the problem was within the ESXi host rather than the network.

After attempting to troubleshoot TCP/IP stack on ESXi, we discovered that ESXi troubleshooting tools were quite limited. Naturally, reaching out to our vendor for support was our next step. Our team handles all of our clients sensitively, but ensuring that the software of a healthcare organization remains operative is absolutely crucial! So when we didn’t receive the urgent support we would expect from a vendor in this type of situation, we had to take matters into our own hands.

Though ESXi TCP/IP troubleshooting tools was limited we found a resolution. Navigating through the supplier’s KB and bug reports ultimately led us to our solution.

VMWare Migration Gone Wrong

Michael Petrov — Tue, 18 Dec 2012 00:00:00

We are in the midst of transitioning a hospital from using physical servers to a VMWare vSphere platform. Meditech is this particular client’s primary software. Our team set up proof-of-concept (POC) and ran countless demonstrations to ensure that all systems, including Meditech ran smoothly together.

We were able to complete about 90% of the migration process before reaching a serious problem! The VM cluster began to malfunction. One by one, physical nodes lost connection to public networks. There were no definite patterns that could unveil a concrete reason for the glitches.

Considering that the old switches operate on ISO the 100Mbp may have been the cause of the problem. We added new ones in front of the core switch; but still no success. Bouncing the network on a physical server caused it to re-establish a connection. Everything pointed to the network simply being unplugged, but that was not the case.

After performing NIC teaming and attempting to transfer cables into different switches, the host would lose all network visibility. Many possibilities were exhausted but our team is coming closer to a solution.

Complex problems like this one cannot be resolved with simple research. If these issues did not arise we wouldn’t be in business. That’s why we are continuing to address this problem head on, boosting our resources and making sure we are doing everything we can to provide 24-hour support for our client.

I will continue to share updates on our progress.

Corporate Mobile Security and Compliance (continued)

Michael Petrov — Thu, 29 Nov 2012 00:00:00

Blackberry has always been a convenient tool. Companies like MobileIron are now trying to solve problems that BlackBerry resolved a long time ago.

I found another convenient feature of BB Enterprise server. If you have an intranet website that is available only inside, you can now access it through your blackberry using private IP address. The Blackberry Enterprise server works as a proxy for the BB device web browser. So if your blackberry has visibility to intranet site you can easily brows your intranet systems though your blackberry device!

Requirements for Datacenters – Let’s Get Serious! (continued)

Michael Petrov — Tue, 27 Nov 2012 00:00:00

To ensure that your equipment runs a smaller chance of being affected by a natural disaster, there are even more factors to consider when analyzing the construction of datacenters.

To start, the datacenter should be on high grounds. Communication and power cables should be routed into your building from 2 different ends.

In addition, an analysis should be made regarding your internet connectivity or remote access. If you have your own AS number and are able to peer and setup BGP between multiple providers, you can guarantee an uptime for yourself, but you’ll need to find trusted partners. If not, you should find an internet service provider who has multiple upstreams that isn’t just connected to a “protected ring” or some sort of redundant connection running on the same ISP.

Though our own core network was stable during Hurricane Sandy, we still ran into service quality issues with multiple partners; Internap being the biggest problem we encountered. The latency would grow from 3 to 800ms and some packets timed out completely. I’m sure this was a result of an equipment overload caused when Internap was re-routing traffic. Those cases are difficult to troubleshoot as they come and go. Our clients get complaints from their clients about outages or slowness but because they are inconsistent, they are hard to be caught.

As these types of issues are extremely important, I am constantly stressing to my sales team and our clients the significance of having at least 4 redundant upstreams connected to different peering partners available.

Requirements for Datacenters – Let’s Get Serious!

Michael Petrov — Tue, 20 Nov 2012 00:00:00

Professionals have said “a datacenter is not just an air-conditioned warehouse.”

After the recent Hurricane, I would add – “They are not just the random basements down the street from your office either!” Making sure your datacenter is specifically built to maintain operation in the event of disaster is essential.

Hurricane Sandy proved that there IS a difference between an actual purpose-built datacenter and the random buildings that have been so-called “converted” into one.

We cannot predict weather conditions or future tragic events, but there are many specific factors that should always be considered when IT management is selecting a datacenter. For instance:

Generator locations.
Location of batteries, PDUs, switching etc.
Quality and separation of chases for power and communication cables.
The size of oil tanks. Two days should be considered a minimum requirement.
Availability of office space.
On-site IT personnel above Level 1 should be available to support IT challenges when your own staff is paralyzed by electrical outages or gas shortages.

For East Coast companies, this hurricane was a true test of all DR measures.

Going forward, I will continue to publish my findings on failures caused by the hurricane and what effective techniques are being used against them.

Not Trivial Monitoring Tuning

Michael Petrov — Tue, 06 Nov 2012 00:00:00

A client requested to know if a server was down when they noticed a reboot occur in their system. The first resource we use for this is monitoring logs. Monitoring didn’t detect any downtime. To verify whether or not the server was actually rebooted, our technician logged into the server to check the event log. We discovered that the VM server did in fact automatically reboot.

Our client then questioned how often we ping for ICMP monitoring and why we hadn’t detected the reboot initially. For up and down events we have multiple sensors with two probing cycles. Our normal probing cycle for this particular client is 60 seconds for ICMP and 10 minutes for port probing and heartbeat report. Because it is a virtual server, the startup was extremely fast. The boot up was so fast that it happened within a 45 second time frame; therefore our standard 60 second probing cycle could not have detected this.

Our clients count on us to let them know when we will have any unexpected reboots. We understand the server could go down in the middle of their transactions so any reboots at all are essential. To resolve this issue we implemented another monitoring sensor. Now we are probing if up-time is smaller than the last heartbeat. Having these two scripts will allow us to detect downtime between the 60 seconds of the first monitor, enable us to check log changes and alert us every 10 minutes.

This case proves that out-of-box solutions may not always work and life forces IT teams to constantly modify and improve their operations and techniques to address day-to-day business needs.

Corporate Mobile Security & Compliance

Michael Petrov — Thu, 30 Aug 2012 00:00:00

I think it is interesting that when I go to use my Blackberry, I have mutliple individuals staring- people I do not even know commenting on why I still use it. Eve
So here is my reason... My interesting point i would like to address.

Blackberry (BB) has all necessary security features and compliance tools. (Some auditors may disagree but you can easily setup framework around BB servers that would satisfy them). BB is still used by some organizations as the ONLY mobile device allowed and I agree with them for numberous reasons.

People jumped on Droids and iPhones because they are "cool toys". Then executives convinced IT to support them without thinking of the big picture. It can be relatively easy to achieve as everybody supported email syncing without a complex BB server, but now the whole security concept has been totally destroyed and has given life to companies like Good Technology, MobleIron, Air-Watch and others who don't even want to support BB. But if you look at what they do - it is exactly what BB server already has done.

It is unfortunate for Research In Motion to lose such a huge marketshare because of these nice gadgets. I am sure that they were counting on their enterprise tuned infrastructure but execsutives didn't think that way.

But here are a few things that makes BB an attractive item for enterprise security:

1. BB server is still a good security framework. Other unified security platforms are trying to achieve the same results for iPhone and Droid devices.

2. Full keyboard. I still like it. Sorry but I still do.

3. Much less useless applications. Your device is for work. It has everything for work. Please tell me what it doesn't have.

Yes, there a big minuses that I have to admit:

1. The device is worse then its competitors.

2. Buggy. Still buggy. Still have to reboot from time to time. In my case it is bluetooth integration with my car system. It freezes after every 5-6 calls.

3. A horible web browser. This is the worse thing in the whole BB framework.

I am not trying to convince people to stay on BB and there will be a day when I think I will have to switch my device that have been my business live for so many years to something else.

I was just thinking about how technology compete and win markets and how IT groups follow and should be able to adapt to trends.

Datacenter Failures- What to consider

Michael Petrov — Tue, 07 Aug 2012 00:00:00

A CIO/CTO should not be complacent listening to marketing/sales pitch about the “cloud” availability, global clusters etc...

They have to ask the questions: What will happen if one of the datacenters goes down for a day, an hour?

What if the whole city is out...

Every company has its own tolerance for global failures and these needs must be accounted for during the solution-building process at the beginning/ designing stages. Companies should be weiry to rely solely on the selected “cloud(s)” availability.

We have seen numerous cases, two of which are listed below. Check out these links.

Because of these issues and many others, I am always questioning clouds hard on what they bring to the table in savings against what the enhanced risks are that could cost my clients money in the future.

So... Until someone can show me different, I encourage being prepared and understanding all of these risks/challenges before implementing any solution.

We saw Amazon going down for hours recently and should know that technology does break!

Amazon Cloud goes Down:

http://www.forbes.com/sites/anthonykosner/2012/06/30/amazon-cloud-goes-down-friday-night-taking-netflix-instagram-and-pinterest-with-it/

Datacenter Fire in Calgary:

http://www.datacenterknowledge.com/archives/2012/07/16/data-center-fire-disrupts-calgary

Log Scanning. Why People Parse Splunk

Michael Petrov — Fri, 03 Aug 2012 00:00:00

I don’t know why people are parsing Splunk. I would never think that such a trivial IT task as log scanning would be turned into a business.

Months back, I spoke with CIO of Splunk and in his terms, Splunk is the " Google" of logs.

It takes a few hours if not minutes for a smart sys/admin to code a simple log scanning script. Someone would say – the devil is in the details – yeah, what about our compliance reports!!!

Here is my response – the whole notation of using log scanning solutions for process monitoring and compliance reporting is wrong.

Here is what Digital Edge does:

EMS (enterprise management system) -centralized IT system that collects monitoring facts from all the systems. Those monitoring systems include log scanning tools.

As some business processes are complex and the monitoring and proactive response is above just finding an event in the log and alert about it; Our system has unique features such as:

a. If an event happened by a certain time

b. If an event DID NOT happen by certain time.

c. If a sequence of events were correct, each event started and completed by its certain time

So, from our position is that if you really want to monitor business processes, those monitoring abilities have to be built into the processes.

We have great examples when IT guys are saying: oh, don’t worry, we can catch any failures, if process crashes, we would have entries in the logs and from the logs would be picked-up by the “Google of the logs” and we will be saved.

Then a little bit later they go: hmmmm, the process didn’t start and we didn’t pick it up as it was no errors…

And later: hmmmmm, the process was just hanging not doing anything and Splunk didn’t pick it up as it was just hanging…

Here is Digital Edge's vision – do not monitor IT systems, programs, components. Monitor business processes and their outcomes. We have seen many cases when all components were running perfectly but the system still didn’t produce expected results on time.

Makes you think...

Monitoring… What really matters.

Michael Petrov — Wed, 01 Aug 2012 00:00:00

This was a long pending blog but the guys below cover most of my ideas. But I have some more: SEE Original STATEMENT

What are we really monitoring?...

Do we really care about CPU utilization? Or Memory. Disk – yes, we do care as it can just stop processing. But more of the information are indirectly related to what we really want to know.

What we really want to know “How the system works for the business”. I call it “business experience”. If business experience is fine, end user or end business process flows feel good, who cares about CPU and Memory and all other things. They could be reviewed once in a while to see if we are really frying that CPU and maybe we are on the edge of the performance. But otherwise if business feels good, we should be OK too. I know I am simplifying things but still, you should go down to simplicity to see the problem.

Our as IT biggest problem that we are measuring technological aspects of IT trying to correlate them to business experience. Yes, it is relevant but we as IT often think that we are smart and can monitor “everything” so when something out of that “everything” is out of a norm, we should react as the business experience may suffer.

Being too long in IT I don’t think that I am too smart anymore. Not because I don’t know lots of things. I know that I know a lot but I don’t know a lot too. Also the complexity in the IT makes that “everything” almost unmanageable. Sometimes a situation when 2 or 3 parameters that are all individually in check produce conditions when business experience suffer. Sometimes you think that you monitor everything but one parameter was not included that would indicate a problem. Then the business experience would indicate “we stuck” but IT would see that everything is OK.

So my thoughts are that we really need to be rigorous and say that we have to monitor business experience and correlate IT sensors to it. Why do I say rigorous – because it is hard. There is no easy solution to this. No “WhatsUp” or OpenNMS. WhatsUp is detecting if it is UP. It doesn’t check if “it” is up AND PRODUCING. What if it is UP but NOT producing?

So I think most of the IT managers just look away from the real problem as there is no out if box solution and real monitoring is hard to implement. But in fact there is a solution, it is not easy, there is no wizards or magic one step deployment. There is work involved but the solution exists. Check our EMS system. This solution helped us to deliver 100% clients’ up time for many years.

Unified Storage Showdown: NetApp FAS vs. EMC VNX

Michael Petrov — Tue, 24 Jul 2012 00:00:00

After reading the recent article entitled “Unified Storage Systems showdown: NetApp FAS vs. EMC VNX", here are a few points I would like to make:

The author says that there is a difference between EMC approach and NetApp approach. While EMC is using Auto Tearing software logic - NetApp’s strategy is to use Flash Cache: NetApp has a Flash Cache PCIe-based solid-state storage device that speeds performance on specific applications, such as data warehousing. It also supports SSDs in the array as other storage vendors do. It is not the same thing and one does not replace the other. You can also create Flash Cache with VNX using SSD disks and use them as a shucks to absorb I/O spikes. I did go over NetApp documentation and should say that NetApp just doesn’t have auto-tearing feature like EMC does.

Both companies are bringing up little features, differences in the ideology. I can put it simple – one was going from SAN to NAS – another one from NAS to SAN. I really like the way Tenaja Group was quoted: It is not relevant anymore.

Another point – there is so much noise about Unified platform – file/block, it is really SAN and NAS in one box. In a lot of cases EMC is used with a combination with VMWare or something like this. Rather than paying for VNX data movers, I would just create a VM “data mover” NAS using either Linux or Windows server. It would be high available through VMWare availability mechanisms and there would be no difference between VNX data movers and that VM NAS server. I can also say that Linux or Windows NAS would be better than VNX data mover. It can implement SFTP that VNX doesn’t have out of box.

There is another strange advantage of NetApp – the fact that they can scale up. Just to uncover “complex” things that usually are hidden a notion of “complexity”. Both are just a SAN and a NAS on the top of SAN. When the author is talking about the clustering – is he talking about scaling out SAN controllers or file movers. If this is just a file movers I would say – who cares. I would use any virtualization platform to create a small VM NAS instance. If you want multiple of them – you can use multiple of them.

Link to original article:link

IT complacency.

Michael Petrov — Fri, 06 Jul 2012 00:00:00

IT complacency.

Digital Edge does not focus in this area of it; however when people ask what is the difference between regular IT companies, and Digital Edge, I find myself describing it best with this analogy, normal companies support, maintain, and fix regular cars. Digital Edge supports, maintains, fix Ferraris and Lamborghinis of the IT world. In reality Digital Edge deals with the most highly complex issues of the IT world, for example, huge high capacity and performance servers and storage, enterprise class networks with high degree of failure resistance and security.

Even though we deal with servers, today’s blog about desktops and “local” IT problems and the headaches associated with these issues. As a warning to “local” IT groups that support users’ desktops – DNSChanger Trojan is still a problem for many companies.

This Trojan comes in combination with a rootkit that makes it a very hard task to fix, and it is always better to re-install affected desktops.

Rootkit is a type of computer infection that gets installed over OS vulnerability, and does everything to hide itself from antiviruses or eyes of the system administrator. We have dealt with rootkits in the past for one of our clients and saw how well it hides itself. It completely took over OS file system functionality and made sure that you cannot see it using ANY programs or a tool that uses OS I/O functions (any iosys.h library). To really see it and kill we had to write a custom program that would look directly at the disk. The stealth virus was hiding specific file names and extensions from tools like DIR or Explorer as well as antivirus, also the rootkit was specifically attacking McAfee antivirus. The virus implanted itself in the OS so it became not separable from the OS. We spent hours fixing the server.

I can write a book about my personal experience of fighting with viruses. Starting with writing my own specific multiplication and propagation mechanisms and stingers for specific viruses in the DOS and Windows 3.11 era and finishing with nova days stealth technologies, injections and IP stack vulnerabilities; today’s subject is DNSChanger.

So there are more than 300 thousand infected desktops running in the corporate world. Some security organizations report that close to 12% of Fortune 500 companies are infected.

We all do have problems and we can get caught by bad circumstances, user or sys admin mistakes, but to run infected systems for months is beyond me. A good system administrator should sense “something wrong” with a desktop or a server. In Digital Edge case I can’t even say what this sense is; but any system administrator should know that there is no smoke without a fire. So if there is something suspicious – dig until you find the problem. There must be a scientific explanation to everything. There are no shamans in the computer world. Computers are scientific objects.

So to the attention of all system administrators, please make sure you are not infected by DNSChangers. To check if a desktop is infected visit this site: http://www.dns-ok.us/

If it is, reinstall it. If you want to have a fun, you can play with rootkit tools from companies like Kaspersky Lab root kit tool TDSSKiller, This brings me back to the days when I had the honor to meet with Eugene Kaspersky back 1980 in St. Petersburg. He was the first virusologyst in Russia by that time.

Implementation and Approach to a New Project.

Michael Petrov — Mon, 02 Jul 2012 00:00:00

Background:

Imagine a client with a need for shared NAS storage, specifically speaking CIFS share – with a total need of 4T.

EX.

A client is actively storing “working” files on the share, and those files are actively accessed for 6 months. However, the uses of these files are specifically used only for look-up, and in other rare cases.

Now, as many companies are, this client is very serious about its SLA's emphasizing that these files and the data mustn’t be lost. Additionally, the client requires the ability to be able to recover files after corruption, accident changes, deletion, or any other operational mistakes that its employees may cause.

Utilizing previous enterprise storage – EMC Celerra - technology was put in place...

For a client, we created a terabyte of “hot” storage using fast SAS disks. This would save money by not putting all 3T of data on fast disks. We also created 3T's of SATA storage to keep files that were older than 6 months.

The following maintenance/backup jobs we setup:

1. We created a Powershell script that would move all 6 months and older files from “hot” disks/volumes to “archive” disks/volumes. The script was also removing WRITE permission from the files for all users in the client’s organization. So archive files could not be modified.

2. We ran an incremental backup using Backup Exec on the “hot” volume.

3. We created a monthly full snapshot of backup on tapes that would be pulled and stored offsite.

4. We also had a standby Celerra SAN where we used CIFS to CIFS mirroring to accommodate requirements for disaster recovery.

Most of the requests that we were getting for the “hot” volume were focused on finding and restoring overwritten or a lost files.

To do so, we used incremental backups to find a trace of the latest good version of backup files in question. But of course, we did have limitations, allowing only up to 3 hour windows, as we were doing incremental backups every 3 hours.

The Problem, Or Problems I should say-

1. The backup window was 3 hours, with full incremental backups taking up to one hour. Reason: The volume contained more than a half of millions of folders and more than 5 million files (Do not ask me why...). And Yes – we did use VTLUs and NDMP and Celerra snapshots.

2. From time-to-time the client was changing the structure of the “hot” volume directory tree, and those changes were not and could not be propagated to “archive” volume. Then when a client employee wanted to find the file in the “archive” the person could not because he/she was looking for a correspondent to the “hot” volume place and the “archive” tree would not sync. This created unnecessary requests for restorations and troubleshooting. Therefore, we started logging every file with archival progress; which then required the client need a GUI to be able to look-up files to see where the actual files were archived.

* In every single case we were able to find the file in the archive and point to the directory structure change.

So the technology setup using EMC Celerra worked but had its own limitations. Those limitations put the client on a project to find “archival” software, forcing the client to start interviewing backup companies and in-turn got dragged away from the original concept by so called “solutions providers”. It certainly created a situation when we felt that we would have to offer something better.

Proposed Solution:

Since the client moved to VNX platform we offered the following solution to address all limitations.

1. To create one big disk pool, containing SSDs, SAS and SATA disks to produce a total space of 4T. 1T which would consist of fast disks and 3T's of slow disks. It is not a huge expense as the disks are much bigger today. Then, we put FAST technology this pool so that the EMC VNX could move data between fast disks and slow disks automatically. By doing that we solved the archiving problem. If moving between fast and slow disks is done on the background, there is no need for the archival script. We could keep all the files in one volume and did not to need archive anything. If the client were to change the folder tree structure there would be no need to synchronize it. Then we analyzed the actual usage of the backups we realized that those backups were actually a version control. The problem was that the backup was done to cover for the lack of version control. If we had something like SVN or SharePoint or SourceSafe, we could have easily avoided doing those incremental backups. So for the “backup” solution we offered the following, providing regular EMC snapshots. When the client needed to restore a file we would mount a snapshot as a different NAS letter and give access to the particular user so that the user could look at how exactly data looked at the “suspected” timeframe. We could then make as many of those snapshots as we wanted. If we have enough storage we could keep the snapshots for a period of a whole year.

After introducing this solution the client’s CEO and CIO; both agreed that this was the way to go. We are currently at the stage of planning for this implementation.

We will be updating on the development and results of the projects as they go.

A blow for Storage centralization/ consolidation fans…

Michael Petrov — Fri, 15 Jun 2012 00:00:00

It may sound that I am against enterprise storage, but everything should have its limits and safeguards. It is petty that because of cases like the one I will reveal below, with such nice architectural design and engineering comes such a devastating blow, throwing technology evolution under the bus. It hurts a lot, when for the last 5 years a 100% uptime statistic holds true, only to be knocked down to 99.9% following one major event. Here is the time line of events today:

1:30pm – an email from Storage Vendor

Hello, this message is to communicate a potential issue on your Storage Array. * Just released- an ETA regarding a potential code issue on systems with iSCSI attached hosts. The SP (storage processor) may reboot after 248 days of runtime. A timer overflow condition within the operating system may occur after approximately 248 days of runtime. If this occurs, iSCSI based network traffic can be interrupted. Depending on the length and severity of the interruption, the storage may interpret it as non-responsive driver software and initiate a SP reboot to clear the issue.

1:46pm - The storage system received too many events in this poll cycle to display here. Contact your service provider to determine if any storage system error events occurred that you should address.

“Jun 13 13:45:21 2012 CS_PLATFORM:NaviEventMonitor:CRITICAL:4 CLARiiON event number 0x7403 APM000000000 SP N/A SoftwareRev 7.31.25 (0.84) BaseRev 05.31.000.5.502 Description Storage Processor (SP B) is faulted. See alerts for details.. ExtCode1=Error; ExtCode2=VNXSISPA.

Time Stamp 06/13/12 17:45:11 (GMT) Event Number 7404 Severity Error Host VNXSISPA Storage Array APM000000000 SP A Device N/A Description Standby Power Supply (Bus 0 Enclosure 0 SPS B) is faulted. See alerts for details.

Time Stamp 06/13/12 17:44:57 (GMT) Event Number 2580 Severity Error Host VNXSISPA Storage Array APM000000000 SP N/A Device N/A Description Storage Array Faulted Bus 0 Enclosure 0 : Faulted Bus 0 Enclosure 0 SPS B : Removed

Time Stamp 06/13/12 17:44:43 (GMT) Event Number a23 Severity Critical Error Host VNXSISPA Storage Array APM000000000 SPA Device SP B Description Peer SP Down.

Time Stamp 06/13/12 17:45:11 (GMT) Event Number 7403 Severity Error Host VNXSISPA Storage Array APM000000000 SP A Device N/A Description Storage Processor (SP B) is faulted. See alerts for details.

Time Stamp 06/13/12 18:04:40 (GMT) Event Number 71274001 Severity Warning Host VNXSISPA Storage Array APM00112300499 SP N/A Device N/A Description FLARE_CPU_WATCHDOG: Flare not rescheduling

00 00 0c 00 02 00 34 00 d3 04 00 00 01 40 27 a1 01 40 27 a1 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 71 27 40 01 66 62 65 5f 73 68 69 6d

Time Stamp 06/13/12 18:26:34 (GMT) Event Number 2080 Severity Error Host VNXSISPB Storage Array APM000000000 SP N/A Device N/A”

3:05 pm- FULL SYSTEM crash.

Down for 40 minutes, the storage failure affected: 4 node Oracle RAC, 3 Node SQL Cluster, 2 node Exchange cluster, and 8 nodes VMWare cluster running around 60 virtual

Now… If you are a CTO or IT director you may want to question these manufacturers, especially when they preach consolidation and centralization.

It is not that the technology is bad but… we still cannot trust it 100%. This is why we implement SANs in DR for our clients.

Advanced Microsoft Licensing Configuration for HARDER security.

Michael Petrov — Fri, 08 Jun 2012 00:00:00

Real enterprise class business requirements in a lot of cases are much more complex than a simple IT solution that you can purchase from a single vendor. Sometimes a simple business case would require a non-trivial IT solution approach; but something that may look simple will require considering lots of simple details, and have small impediments that are hard to overcome. Solving such cases require either deep knowledge of technology or extra money to cover for the lack of knowledge.

Consider this situation,

A client runs a Microsoft domain as its single user security and policy control mechanism. One of the executive’s requests to create a virtual desktop that would be used specifically for his needs. The security of that desktop should be outside of the domain so no domain admin can look in the information on this virtual desktop. The desktop should be highly available, and as data is critical it should have either tough backup policies, or redundancy. The client also is heavily invested in VMWare clustering so a natural solution would be to create a virtual machine on VMWare cluster and the HA problem would be solved.

The problem of having a desktop outside of the domain is that the licensing for the remote desktop is in the domain, and you would have to purchase separate licenses for the server if more than one user wants to connect. In a case we ran into, this executive notified us that he wanted other top managers to be able to lookup the data with different privileges. We certainly created appropriate ACL’s for the access control using local users but new licensing had to be purchased, as we could not reach the domain licenses.

Here is how the solution worked that allowed us to avoid new license purchasing:

“After rebooting, I added a terminal server role into the configuration. There you can select the licensing mode; which in this case was selected by user. From there you can point to a licensing server; Adding licencingserver.domainname.local. Once done, I could not validate the licensing server due to lack of authentication, since the virtual desktop was not on the domain. In the licensing diagnosis window you can select an option called ‘provide credentials’. I entered my domain admin account info and it successfully established the connection.”

Now, I know that $85 per RDC license x 5 users is not considered a huge savings for most companies; however… the POINT is that it is a Savings. And that small savings in this nature bundled together can alter IT budgets to be put to much better use, Don’t you think?

At Digital Edge, I am always enforcing this type of thinking into my Engineers and Project Managers as STEP 1, 2 reasons…

1 to show the client our level of commitment and 2 to instill options that help conserve IT spending for more critical initiatives.

Digital Edge ticketing system.

Michael Petrov — Fri, 01 Jun 2012 00:00:00

Digital Edge is ready to launch its third generation ticketing system. Since its original version, we have always been committed to the development of our internal ticketing system. My vision was that this type of internal design would pay-off in the future, and I found myself to be correct. The development of a ticketing system, the flexibility of design and the necessary add-on features that can be intergrated as you see the need, really make Digital Edge stand out to its clients.

I know to most people, a new ticketing system is considered a simple task, but… realistically when you consider the implementation involved, I would definitely disagree. Technically, a ticketing system should be looked at as a core component of a business, where requests are registered and reports are sent out for status updates.

For our clients, we use our design to customize our client’s needs, as all businesses are different. Response time is crucial; therefore we make sure our ticketing system is robust and transparent.

Also- when IT organizations handle multiple amounts of tickets it becomes not a trivial task to control SLAs on those tickets but more of a timing of actions that plays a huge control role in the process of delivery.

From the first days of its existence Digital Edge a high end IT Company was committed to develop its internal systems, and we know that this strategy pays off as everything is integrated in our engine by design.

Here are some ideas and features that we built into our new ticketing system.

1. Clients are able to manage tickets through phone/PDA.

2. Complexities of tickets do not allow simple emails to be used as a centralized ticketing system, so web based database driven system should be heart of the ticketing. It also should receive messages from phone, web, and email sources and store them in one central database for further handling.

3. All of our clients love simplicity of our previous ticketing system. You send an email, and in reply a ticket number is received. So we had to keep this simplicity; adding more control.

4. We have to be able to time communication messages. For example, if a client replies to an email from a support team we need the support team to handle that reply within a certain time period, so simple email would not work.

5. We need to categorize tickets so we can assign different SLAs levels to them.

6. Tickets should be closed by clients.

Here is the logic that we implement:

1. As the original system, a client doesn’t have to login, remember passwords etc. to open a ticket. In emergency situations, clients cannot spend time to open tickets through web interface so they just send an email to support@digitaledge.net

2. We receive the request and assign ticket following properties:

a. Client Name

b. SLA class (Production critical, Non-production critical, Change request, General info, General troubleshooting, Programming debugging, DBA request etc.)

c. Based on SLA class we setup a delivery time. It can be: ASAP, has to be done by certain time, have to start at certain time, have to start and finish by certain time.

d. Difficulty level (for our company performance reviews and statistics).

3. We send this info in the reply email with the ticket number and a link to the ticket so the client can see the ticket in read only mode. The link has auto-login functionality. So the client needs to just click on it.

4. When client clicks on the link in the ticket, he can request a re-classification/escalation of the ticket, or provide more information.

5. If the client doesn’t have a browser, he can just forward this ticket to ticketreply@digitaledge.net and the system will catch this forward and update ticket record accordingly.

6. When we receive a reply from the client, the ticket info gets updated. Our engineers must acknowledge the response. If an engineer doesn’t acknowledge the client response in 5 min, our monitoring group will escalate the issue to the next engineer or the management.

7. When a ticket is done, the engineer changes its status to “Close Pending”. The client receives an email with 2 links – one to close the ticket, one to provide more information. When a client closes the ticket he can provide his feedback to management about the ticket handling.

8. If a client does not close, he receives multiple reminders that the ticket will be auto-closed. The ticket closes if the client does not reply. The client can re-open a ticket any time.

With implementation of this logic we will get:

1. Better control on timing of the ticket execution.

2. Better control over the workload and schedules.

3. Get a feedback about ticket closing.

4. Assess performance of the employees based on the difficulties of tickets.

Digital Edge is committed to constant innovation to bring value to its client. We hope our clients will quickly get used to it and will love using it in near future.

Your business operates the way its systems operate?

Want to patch Dell Firmware on the server running VMWare – BUCKLE UP

Michael Petrov — Fri, 25 May 2012 00:00:00

We know sometimes servers misbehave, and hardware manufacturers issue patches to their BIOS or firmware. Most of those patches are packaged to be installed through MS Windows or Linux. Dell does not package patches to be installed through VMWare ESXi.

Dell 11G+ servers have a new feature called Lifecycle Controller which is an embedded app that you can do maintenance through. One of the features of this app is the ability to update firmware through FTP or local media. However in one particular case we were not able to use FTP, and had issues using local media that even Dell Support could not assist. Normally, when updating firmware on Dell servers, you have the ability to patch with Windows, or Linux OS. However, ESXi is different and there is nothing that you can run through the service console.

In order to resolve the issue we had to boot up the server from Linux LiveCD and then patch firmware through Linux. It was a very silly problem as well as strange that Dell did not package upgrades through ESXi, especially knowing that most companies today use type of hypervisor on hardware.

You would think manufactures would prepare hardware related fixes for different flavors?

Making Sense of Cloud.

Michael Petrov — Tue, 22 May 2012 00:00:00

Back in August 2011, Gartner stated at their Summit in Sydney ... " by 2016, 20 per cent of all business organisation’s IT will be purely Cloud-based was put to delegates and a panel of IT gurus. Roughly 70 per cent of an audience of around 250 agreed with the prediction. Here’s some of what the panel had to say."....
http://www.cio.com.au/article/396899/compromise_likely_cloud_computing/

It is funny that an organization that pretends to be an IT expert is more marketing/sales and much less technology/engineering driven. As a reputable IT publication they should be ashamed of such broad and loose statements. They really sound to me like "New Generation Chooses Pepsi!!!".

Cloud-Based is such an overused and misused term that nobody really knows what it is. Let’s get back to engineering and try to make some sense out of what was said.

- All new IT initiatives including Cloud-based infrastructures are based on virtualization technology.

Virtualization technology allows the use of power from one physical computer to be divided by multiple logical virtual computers. So in a simple case you can have one IT division take care of the hardware level with the virtualization platform, while another division can concentrate on applications that run in virtual servers; totally abstract from the physical design.

Therefore, IT can do it on its own or TRUST someone like Amazon Cloud or anybody else to provide the hardware platform for your application hosting virtual servers. The keyword here is TRUST. If you trust someone to run your virtual instances inside their infrastructures it means you are going into a SHARED physical processing environment.

By nature, Digital Edge provides services to Enterprise class IT organizations and their mission critical business services. One of the biggest concerns for these Enterprise organizations is the control of RISK; And today it is just not feasible to manage risk in shared cloud environments. I agree that a lot of small non-mission critical businesses will find public clouds very appealing. But for serious mission critical services it is not acceptable.

Here are 3 BIG problems with cloud-based processing that drives my skepticism about the Gartner statement:

1. Managing risk in SHARED environments.
Lots of cloud providers brag on their SLAs but for any serious IT executive it should sound like – “I will pay you pennies for any outage but you cannot sue me.” We receive so many requests such as - do you provide SLAs? Yes we do! But how much? One of my friends was complaining that one of those SHARED environments brought their infrastructure down for 22 hours and offered $78 as the SLA payoff!!
Any serous IT organization supporting mission critical operations is not at the level of complacency when a provider can bring you down. Any unmanaged down time can result in huge loses that IT department would not be able to tolerate.
Many times in my life, IT Managers/Directors/VP’s whose jobs and reputations were on-the-line would call me directly for a private meeting to clearly state – “I understand that you are running a great organization. But… if one of your technicians in the trenches (operations) makes a mistake, a dozen of executives may loose their jobs. So please make sure that you have triple control, please make sure that even though you have that triple control all the guys also know that their decisions will affect lives and financial outcomes of lots of people that they don't even know.”

2. Getting peak performance out of the hardware.
IT is easy for cloud providers to say that we will give you as much power as you want. In the end of all debates most of the clients of vendors like EMC, 3PAR, Compelent, Fujitsu are business clients, not cloud providers. It means that business require not just capacities but most of the I/O that they can get. If a client requires to process a few hundred MB per second, there is no way they can go into a cloud.

3. Security
Sorry for bringing this up to your attention but again, Cloud based is SHARED. You have to trust your provider not to just operate your hardware with no down time but also be sure that this provider can keep your data secure. When you buy a locker box from the bank, you can get in and see how it is organized. You have one key and bank has another key. Even though you see this, you don't know how strong the wall is behind those lockers. You don't know how those keys in the bank are managed. You don't know if the security staff is trained. A bank would have some insurance when/if they lose your money to pay it back, but when a client loses its reputation for data loss, the cloud-based provider cannot return this reputation around nor can they have an insurance policy for this type of ruin.

So, in conclusion, I do believe that virtualization is a great technology but cloud-based or private is still a fundamental decision that every IT organization will decide on its own and that the cost advantages of shared hardware will NOT be the main driver for such decisions.

Danger of Building Cross Connects

Michael Petrov — Fri, 11 May 2012 00:00:00

We have no idea how connections from ISP or Telco equipment is delivered to our switches, routers or firewalls. The building management may sound professional and confident – “we will deliver you 2 lines from 2 independent ISPs”.
Well how do we make sure that the delivery mechanism will not use a single point of failure? You may spend time, and design your HSRP/BGP architecture to ensure failover and guarantee up time; but do you know if building management spent the same amount of effort? If you have your redundancy figured out, how can you enforce building management to respect your concerns about high availability, and do the same as you?
Well, we saw the BGP mess, during their building maintenance fiber cross connect infrastructure started going up and down.
Suggestion to IT management: Checks with the building on how to cross connects are delivered. Do not trust words and promises.

Wasteful Spending!

Michael Petrov — Mon, 07 May 2012 00:00:00

I am constantly advising our clients and other IT organizations to consult with us before making any decisions. Questions do not cost anything, however questions are the significance to cost saving. Unfortunately in a lot of cases TCO for IT departments has a cost of stupidity component.

One of the examples as I mentioned in previous blogs, Telco relationship, cross connects, ISPs, even though it sounds very simple may be very complicated, and can cost a lot of money.
I don’t want to just give details about an IT manager trying to save money, and falling to the first Telco cold call. Per se when Telco is talking to me, and quoting me on the price, first thing that comes to mind is how much more in hidden charges the client will be paying at the end. I am just curious why clients don’t ask and negotiate with Telco providers to deliver what they are promising.

So normally the deal unfolds like this:
Telco : we have a great promotion. You have your servers in New York and in Boston. How much do you pay?
IT; $3000.
Telco: we can do it for $2600.
IT: OK. Good. I can save 400.
Telco: (delivers their solution on time if you are lucky).
IT: Wow, great. How do I switch now?
Telco: Connect and test. Then switch.
IT: Where are the ports?
Telco: You have to ask building to cross connect us.
IT: Building management, we need to connect to the Telco.
Building 1: no problem. What are the specs for the cross-connection?
IT: 100Mbps, RJ-45, Cat-6.
Building 1: No problem – 260 per month for one run.
IT: wow, and this is on one end. On another end I know that the building management charges only setup fees. I paid once for that old cross connection so I will pay the same one time?
Building 2: we stopped doing one times charge and start billing on a monthly base, but the good news is that you don’t have to pay one time charge anymore. The charge for this connection is 220. But if you want, we can get you a fiber cross connect so in the future you can push more.
IT: Good idea. Let’s do it.
Building 2: Done – here is your fiber hand off. Now you have to either use a media convertor or have equipment in your LAN that is capable to receive fiber
IT: I don’t have a media convertor, no fiber capable equipment. There will be more investment in hardware…

Certainly it sounds absurd; but this is how it unfolds. This can go on and on during the switch over, and agreement with the building.

So my question to the IT community: Why do most clients let Telco say: connect to us. Why shouldn’t they be accountable to provide a full solution, and deliver it right into your rack?

The second question to the IT community: Why are providers allowed not to disclose all additional charges and related expenses?

Storage Failure- a Huge Success?

Michael Petrov — Mon, 30 Apr 2012 00:00:00

Yesterday Digital Edge encountered an EMC CLARiiON CX-4 failure. It was a major component failure rather than just a disk.

Centralized enterprise storage is the most critical part of any infrastructure, and any Enterprise IT engineer would have an unpleasant feeling, a feeling that would be described as a physical pain concerning storage failure. In certain situations with a bad design "everything" can go down, and who knows for how long and how much data will be corrupted or unrecoverable.

Serious storage failure in most cases ends up with some financial consequences, job loss; money and/or reputation loss etc…

So in this case, EMC had an SPB controller CPU failure, half of its internal architecture. It is as if a human would have had a stroke damaging the brain’s left hemisphere, leaving the person unable to move at all. However not in our case we were up and running, here is why…

The communication from servers to SAN was configured with Multi-Path where volumes were load balanced between SPA and SPB. Each volume had a preferred path with the ability to failover to the failover path. During the failure all volumes paths with SPB preference were migrated to SPA seamlessly and applications didn’t even feel a blip.

This is a result of a good design for Fault Tolerance on each single component the SAN – Servers, Fiber, Switches, SAN itself. In theory we would feel service degradation, but didn’t notice any slowdown as the CLARiiON is a very powerful device – at least for the client’s current needs.

As for recovery, we received the part from EMC, and were ready to fix the failed component in 2 hours. As for industry standards it is a very quick turnaround for hardware manufacturer, its support, and supply organization.

Lastly, I am sure that the failed CPU was not manufactured on US soil and rather produced somewhere on the other side of the world. However shortcoming of the manufacturing quality was greatly compensated by great device, and infrastructure engineering.

Oracle Deployment 11G RAC to New Hardware

Michael Petrov — Tue, 24 Apr 2012 00:00:00

No matter how you plan or how well you’re prepared, upgrading core production components tends to be a difficult task. This is exactly the reason why people are running on older software/hardware for years trying to avoid the upgrades, even though Hardware and Software vendors are pushing hard for upgrades threatening clients that they are not going to be supporting their products any longer.

There are multiple challenges in upgrading to a newer version, but all comes to three major questions.

- Will applications work with the new version without any new bugs or outages?

- Will new version perform the same way or better?

- How are we actually going to do that?

Digital Edge is helping one of its clients move from Oracle 10G RAC to Oracle 11G RAC. Oracle RAC is powering most critical business applications of the client, and is central repository for all its data.

First let me explain WHY we had to do it even though I think it is a bad move.

Why we had to do it:

a. We purchased a new EMC VNX SAN, and originally wanted to migrate Oracle ASM and Oracle cluster ware on the new EMC SAN.

b. The client wanted to get more performance out of Oracle, as they started to bottleneck CPUs. We were thinking about new hardware and started talking to Oracle.

c. Oracle suggested that to be in compliance with licensing we would have to change the CPU architecture. Here is the quote:

“Your current production cluster is based on 2 x R900 servers each with 2 x Quad Core Xeon CPUs each CPU running at 2.4Ghz

The recommended architecture is based on a primary 2 node server cluster for prod and a 2 node cluster for DR.

Each R710 is configured with a single very FAST quad core processor, this was specifically recommended and laid out in the
presentation so you Oracle License costs could be managed, if this was not configured in this manner and you continued
to run servers with 2 x Quad core configuration, you would be looking a license exposure well above $700K.”

So now we ended up with 2 Node Oracle 11G RAC in production, and 2 Node Oracle 11G RAC in DR with a plan for physical Data Guard between them.

Why it is bad:

a. I feel that the servers that we run on are pretty good. Those are Dell 900 with 2 CPUs. We had to get 2 CPUs vs 4 CPUs available to be in compliance with Oracle license.

b. Oracle 10G is on production boxes across the industry for a long time. Has its glitches and problems, but it is a known devil.

c. The main app and all supporting and secondary apps are tuned for 10G performance.

d. In attempt to comply with Oracle new license model we had to move to more powerful processes but have less processing power counting cores.

Here is what we have to do now:

1. Find hardware to deploy testing environment in parallel of PRODUCTION and QA environment that we already have; keeping QA for new releases of software. We also need to have enough hardware to produce adequate load to see how new implementation is configured.

2. Make sure that business users will not confuse TEST environment with actual PRODUCTION as it would look almost the same.

3. Make sure that we run all business cases but at the same time performing stress testing measuring performance base lines.

4. We have to plan for switch over process and measure the time for the switch over process.

5. During the client tests we have to gather base line stats and tune statements per new 11G optimizer.

As everybody knows the hardware cost is more about price of power. Electricity cost has increased in the last two years. So to bring parallel infrastructure up for a month or two, would require additional power, processing capacities, etc...

“This is a great application for public cloud” would jump someone who is relatively new in financial processing – not so fast. The data has to stay secured in the secured perimeter. We cannot just build client servers somewhere outside; also latency between cloud servers to Oracle would invalidate any real stress test.

Certainly our virtualized application layer helped a lot. We had enough resources in our VM clusters to build most of the parallel processing. However we still had to power up a few older physical servers to compensate.

Lastly even with strategic planning this big upgrade put a stop on multiple depending infrastructural project for at least month or two.

By the end of the project I will try to calculate the cost of pre-production test effort to have an idea what the technology upgrades actually cost to the end client and if the whole thing is worth it.

How To Deal With Telco Providers When Ordering Cross Connect For Your Business

Michael Petrov — Tue, 07 Feb 2012 00:00:00

One of the essential and most typically considered “basic” aspects of Enterprise IT is Telecommunication cross connections. For most IT management these connections are “simple”, but these “so-called simple” and crucial IT components have the potential to become the largest environmental pitfall. With hundreds of ISP’s (Internet Service Providers) or Telecommunication vendors, these critical connections become the victims of quoting battles, similar to other IT constituents- the Winner, most likely the glorified “lowest bidder”. However, before selecting the manufacturing vendor that agreed to meet your price-point, some apple to orange comparisons should be made that just might change the adjacent signature to the telecom contract you sign.

First, the BIG PICTURE–

You collocate equipment in a Datacenter and decide to have a private line for speed, security and insulation from the corporate office; Corporate Office- Point A, Colocation- Point Z.

The best situation is to have a Telco provider that has its own equipment in both places (POP – point of presents). This usually drives down cost and implementation time. Once an agreement is made, the vendor selected must configure a “virtual” path between equipment located in the POP of location “A” and “Z”.

From there, the scenario usually goes as follows- “We are ready, we have 2 ports setup, one in Point A and one in Point Z, now connect the switches or routers”. -This exact moment has to be discussed prior to ordering the line - what exactly will be delivered?

Does the provider deliver hand of right to your switches/routers or have a port for you in their equipment that is in the same location as yours?

To go back to the basics for a minute for those of whom made need a refresh, the connection between Telco equipment and your equipment is called a “cross-connect” and in the most cases is managed by the building. Telco from the building management perspective is a regular tenant.

Two huge factors- connection fees and distance: Telco from the building management perspective is a regular tenant, so building charge fees to capitalize as much as possible. For distance, depending on how far, the building might have to use fiber that would add an extra cost, level of complexity, and failure points. I fiber is involved but you agreed with Telco on Ethernet hand off, then there should be 2 media converters on each end of the fiber to convert FC into RJ45. Those converters are additional cost and points of failure.

Cross connect fees can be seen as a one-time installation fee that could vary from $1000 to $3000 or could be monthly ranging from $100 to $500 per month.

Going into a contract with a Telco and hoping to save $500 without accounting for cross connect fees on both ends might transform a mentality for savings into a position for losing money. Also delivery time of cross connects from the buildings can dramatically effect deadlines. Some facilities are forced to use union workers and may cause weeks’ worth of delays.

It is always better to make the Telco provider responsible for cross connects. They know how to deal with facility managers, ordering procedures, and LOA.

Another problem might arise when a Telco that is giving you a perfect deal doesn’t have its own hardware in Point A or Point Z or both. In these cases a so-called Teclo “swap” must take place. Telco’s negotiate between each other at both points to either purchase “last mile” or swap it for services/bandwidth in other locations. These “swaps” cost money, so the Telco that is the “closer” to the client will always have leverage over other competing providers.

If a Telco partner of your choice is planning to do the swap you always have to know about it. You need to know who are the partners in the deal and what chain of swaps are involved. There are known industry horror cases when Telco partners were finger pointing to each other for problems and end client downtime. There are also cases when the Telco NOC (Network Operation Center) would not even know about the swap and would not be able to find circuit ID’s to push a problem ticket to its swapping partner.

Because of the numerous issues that companies face daily, Digital Edge advises the following:

1. Besides the price look for Telco partners that would have equipment in both desired locations.

2. Do not go with a partner that has more than 1 swap deal. The delivery would depend on the swap third party that you would not have control. Only if there is no other option you should go with a Telco that would require more than 1 swap.

3. Negotiate a delivery mechanism and types of hand-off and cross connect responsibilities. Plan and explicitly specify how cross connects will be delivered and who will be paying for them.

4. Understand each cross connection solution, involved parties and NOCs. Keep records of all circuit IDs, NOC numbers, project manager’s and sales people contacts.

5. Explicitly tell your Telco partner to label each circuit with an extra label for example: “Circuit deliver point to pint line for XYZ inc. Do not unplug without authorization of following people:”

Work with the Telco partner to make sure that the label is clear and help avoid outages caused by human mistakes.

Ask Digital Edge Sales Engineering for advice on your future Telco needs; cross connect architecture, MPL architecture and other Enterprise IT aspects.

Enterprise Storage Selection Methodology

Michael Petrov — Fri, 23 Sep 2011 00:00:00

IT space is always overcrowded by all types of vendors. When a company has a need to purchase a new gadget, there will be hundreds of vendors around IT groups preaching products calling the shamelessly not less as the BEST on the market. No matter what they sell. It is hard for CIOs/CTOs/IT directors to make a conquest choice as we all know that vendor offers are sliced differently and you always compare apples to oranges.

The biggest goal of IT groups is to figure out how to compare offerings on a flat place, how to make them equal – make them all apples or oranges.

This is how we did it selecting a SAN solution for an entry-level enterprise implementation.

1. Narrowing the selection

It is hard to come up with an algorithm of how you would narrow the vendor selection. We can easily contact 20 vendors but it would be too much to compare. As an initial selection, we decided to review 3PAR, EMC, NetApp and Compellent. We, as a service provider, didn’t even go into this selection as it was no difference for us if the client will want to review NEC, HP or many other manufacturers. At this point we just got a list from the client who said they want to compare those guys…

After communication between our client and the vendors, the client narrowed the selection between EMC and Compellent. This is where our work as trusted consultants started.

2. Creating a “flat place”

Here is what, in our opinion, counts when you select storage: capacity, power, features.

Capacity:

Capacity analysis should be broken down by:

Raw offered capacity

Raw capacity is easy to calculate number of disks and their sizes. Naturally, it has to be broken down by the class of disks as faster storage costs more and has smaller sizes. IT should never get tricked by marketing talking points like: “our storage software logic makes it so fast that you don’t need many fast disks”. Do they think we are stupid?

Derived usage capacity

In the past it was very easy to calculate usable capacity. Today storage vendors offer different types of RAIDs, pool algorithms and configurations that have certain limitations. Those limitations create lower physical disk utilization, require more hardware and sometimes result in usable space loss or underutilization. Down the road you may easily hit a wall where the array would not allow you to use RAID 5 with 3 disks and would force you to use only RAID 6 with a predefined number of disks. So we would strongly recommend creating a disk usage map, RAID groups and pools at the time of storage selection.

Maximum capacity

Maximum capacity is the number of disks that offered controller configuration can spin before controllers have to be upgraded.

Power:

It is very hard to analyze power. There are many different methodologies to do that. What makes it difficult is that the same application can work differently on different setups. For example, there could be a big debate on what is going to be better to throw? 10 SSD disks in a disk pool or use them as a fast cache. Maybe it is better to use 5 SSDs as a fast cache and 5 SSDs to add to the disk pool? And if you do that, maybe it will be impossible to use 5 SSD disks in the mix with SAS and SATA disks in the disk pool and we need 6 of them.

Here are the parameters that we wanted to use in order to analyze raw power of the storage:

Total Controller RAM

How much RAM controllers have. (ßI’m not sure what he means here.) SAN controllers are computers. Some of them run chopped off a version of Linux, some of them run proprietary OS but they all use RAM. The more RAM controllers have, the faster they work. There is one huge subject – OS version. If SAN OS is 32 bit – the RAM limitation would always be 4G. 64 bit SAN OS would not have this limitation.

Fast Cache

Fast cache plays a role of shocks absorbing I/O spikes and we feel it is a very important part of a SAN implementation. Some vendors don’t have fast cache at all claiming that the algorithm is optimal as it is. So if you select a car, would you get absolutely the same engine without TURBO or with it if the price is the same? The same goes for fast cache. The more of the fast cache is proposed, the faster the system can be.

Total Ports and their aggregate speed

For sure port availability should match. If one SAN offers FC and iSCSI and another one only one of the options, you would probably select the storage that offers both types of ports.

iSCSI could have 2 speeds: 1Gbps and 10Gbps

Fiber could have 4Gbps and 8Gbps.

In a lot of cases you would hear that if you have a high performance application you should go with Fiber. You would ask if you have 1OGbps iSCSI, isn’t it “faster” than fiber? Not necessarily. The problem is in the frame size and the way data is encapsulated. The detailed analysis of the iSCSI vs FC is outside of the scope of this document and should be performed using application specific data. However, the cumulative speed of all ports could give you a good idea about the power of the system.

Maximum throughput on the front end

Technically it is the total port aggregate speed. So if the array has 4x1Gbps of iSCSI ports and 2x4Gbps of FC, the total would be 12Gbps. No matter what you do, you cannot pull more out of the array without changing or expending the controller.

Maximum throughput on the back end

This is the throughput from controllers to the disk enclosures and disks themselves. The speed depends on the architecture of the array and cannot be easily expended.

Number of IOPS (Input output operation)

It is a very important parameter. This is how many input output operations the array can perform in a second. In contrast with throughput, the IOP can be very short but there might be a lot of them. In a lot of cases OLTP database would be an example of when the throughput is not so important but number of IOPS is something that would affect the performance of the system. Number of IOPS depends on number of physical disks (“spindles”) in array (certainly SSDs are not spindles but have their own number of IOPS characteristics).

As an example – 4x600GBx1500RPM SAS disks would give 1.2T of storage. At the same time 8x300GBx1500RPM SAS disks would be the same size BUT WOULD PRODUCE TWICE MORE IOPS. So database systems would perform better on the second configuration.

Important features

All SAN arrays offer different features and comprising is out of the scope of this document. We wanted to point out features that can affect total cost of ownership.

Power consumption:

As power is expensive today, total power consumption is a very critical parameter. It is linearly dependent on the number of physical disks in the array as one single disk can pull up to 0.5Amp power consumption. So overall it is better to have fewer disks which contradict with the IOPS recommendations.

Maximum number of disks

Some SAN vendors would say that they can spin unlimited amount of disks. There is no theoretical limit; however, there are CPU/memory resources needed to control/spin disks. When the number of disks is going up the controllers will slow down. So it is very important to know when there will be a need for an upgrade.

This table shows the comprising of 2 different SAN arrays based on our methodology.

	Compellent	EMC
SSD 100G	0	10
SSD Storage	0	1T
SAS – 15K RPM 600G		41
SAS – 15K RPM 450G	48
SAS – 15K RPM 300G
SAS – 10K RPM 300G
SAS Storage	21.6T on 48 spindles	24.6T on 41 spindles
SATA – 7.2K RPM 2T	12	7
SATA Storage	24T on 12 spindles	14T on 7 spindles

Fast Cache	8G	200G
Max Fast Cache	64G	1T

Disk trays	4	4

Ports FC	8	8
Ports iSCSI	4	4

Max Throughput on front end	64Gbps in FC + 4Gbps on iSCSI	64Gbps in FC + 4Gbps on iSCSI
Max Throughput to back end	96Gpbs	92Gbps
IOPS Based on Drives in array	Could not get	38,640.00

Max number of disks	150 (estimated)	250 (limited)

Power consumption	2.8KW	1.47KW

OS	32-bit	64-bit

3. Comparing technical details

Based on the comprising table we can see that:

EMC provided better disk configuration.

The max throughput is the same from both vendors.

IOPS could not be obtained from Compellent

EMC provided Fast Cache while Compellent claimed that the algorithm of the software would compensate.

EMC is natively 64 bit. Compellent was offered in 32 bit configuration with an ability to upgrade to 64 bit in a few months when it will become available.

4. Cost

At first, Compellent offered a price close to EMC but was around 3% more expensive. When Dell found out that EMC is winning the deal, at the last moment dropped prices almost 100K down. Our client felt it was not ethical for Dell to do that and still wanted to go with EMC, however showed the prices to EMC for comments. EMC provided additional discount but could not come close to the lowballed price of Dell. The client still selected EMC as they felt that Dell may try to recover the loss of revenue through further charges.

Complexity of cleaning up networks with SAN/NAS environments in case of virus outbreak.

Michael Petrov — Wed, 12 Jan 2011 00:00:00

Sometimes clients do get viruses. There are cases when regardless of all security and antivirus measures computers still get infected.

This case introduced here is interesting in multiple ways.

1. The way the client got infected. We had a snow storm on the East Coast. People could not get to work so we started getting requests from management to open VPN connections to the environment for a lot of people. Before the snow storm we had it opened for management only and assumed that people on the VPN are clean and we were letting them into the secured perimeter over encrypted tunnel without any limitations. We did not consider massive VPN connections from home computers and got a user that brought an unknown “creature” to the network.

2. Even protected servers got the infection because the virus managed to disable enterprise antivirus software and infect most of the network. This was a failure on the part of the antivirus software but I don’t want to mention the vendor name here. Unfortunately we didn’t have time to learn the behavior of the virus and figure out all the details of the infection and propagation.

3. We enjoyed our segmentations of functional blocks so we got infections only on terminal servers and it didn’t spread to database servers, mail services and other heavy lifters. So the segmentation philosophy proved itself again. The way we think is that we should manage each functional segment separately versus practices of managing things centrally. Some admins like to manage one big firewall, one big database etc. Even though there are advantages of minimizing managing points, those architectures expose networks and systems to bigger failures. We do know that technology fail where you would not expect no matter how experienced you are. We all had cases when regardless of planning for failures, redundancy, backups, something still happens that nobody could imagine. So segmentation reduces the risk of global failures localizing problematic components.

4. The client has a huge NAS running by EMC Celerra. More than 8 million files are hosted on one single share and we have 7 shares. When antivirus scans through those shares, it takes more than 5 hours. So infected users could plant viruses back on NAS to the areas that were scanned already. So we were constantly chasing our tail. We also use the same Celerra as a native SAN powering SQL Server cluster, Oracle RAC and Exchange cluster. Those servers and volumes were not affected by the virus because of our segmentation policies.

5. Microsoft response. We were actually very surprised by Microsoft help in this situation. It took much longer for the antivirus software vendor to react so we decided to contact Microsoft. After we opened a PPS ticket we got informed that virus outbreaks are Microsoft priority and any fees will be reimbursed back to us. We got an American support team (not offshore). Very knowledgeable person. He told us why we are having such a problem cleaning up – apparently the virus infects DLLs and EXEs and encrypts them. So decryption of such files is a problem. Some antivirus software doesn’t pick them up because of that. The virus is also polymorphic, so its signature changes from time to time. We have explained the way we found successful in preventing virus spread. The engineer told us that we are on the right path but gave us an official “Microsoft” way – a KB article - of preventing spreading like this.

Besides that Microsoft provided us with two tools. One is for online scanning, the other one for offline. Those tools are not auto-updatable, but IT groups can call Microsoft at any time and request fresh copy of those tools as they are updated by Microsoft team every 6 hours.

Online tool is to scan memory and file system of production servers. However if the server has a ROOTKIT then the tool could be useless as ROOTKITs hide viruses. To clean such server, the best way is to rebuild the whole server. Another way is to shut it down, reboot it using a CDROM disk built with the second – offline Microsoft tool and scan the server while OS on local disks is not loaded in the memory.

Another way to detect ROOTKITs is to share disk drives and scan shares remotely from a clean server. ROOTKIT hides viruses and itself because it is patching OS, change memory management and file system. When you are remote, it cannot hide and antivirus on the remote machine would detect infected files including rootkit itself.

We saw cases when remote scan detected infected file, but when we were on the infected server, we could not see this file at all as ROOTKIT was hiding it.

After successful blocking of spreading and offline scanning of servers one by one we could successfully clean up the whole environment. The most important for our client was the fact that they were not down a single minute as our architecture allowed us to keep the system afloat even though we had to do offline scanning on multiple boxes and viruses and virus scans made some production elements slower.

Lessons learned:

We learned that our segmentation philosophy worked great as we kept production of the client uninterrupted.

We also learned that we really need to limit user’s abilities on the network. The users of this client had too many rights and privileges on terminal servers.

We decided on a plan to upgrade the client’s network topology helping preventing viruses from spreading, stealing passwords or data.

We developed a constructive plan to implement better AD policies, terminal services usage policies, VPN and other.

We also are in the process of rebuilding some elements of infrastructure using virtual private cloud elements. Virtual server architecture will allow us to rebuild servers from snapshots quickly rather than cleaning them up.

It is very strange to see some IT and Financial executives’ decision process.

Michael Nikhamov — Thu, 16 Dec 2010 00:00:00

Here is an example. A company is interviewing datacenters partner for co-location and full management services of their network infrastructure and servers. The infrastructure consists of about 100 servers and 40 network devices and appliances. After providing them with our proposal we found out that we are the most expensive provider.

We could not believe this. Our prices are very competitive. We certainly cannot get our hands on our competitor’s proposals but after questioning IT staff of the organization we found out that our competitors offer the same colocation AND full management price as we propose for colocation only. We were in shock. We thought that full management would lack some services and started hassling for more information. Asking if DBA services are included, if change control is included, if patching is included – the answer was YES!!!

Having 140 devices infrastructure and including full support pricing in the colocation cost sounded insane!!! How can you compete with this??? And more over how can anyone deliver those services in such budgetary pressure??? Should just close our operation and go out of business???

Here is how we do this. We cannot provide full management services if we do not deploy the client in our management platform. It includes device repository, monitoring, knowledge management, ticketing and task routing system, scheduled activities, change control, backup and recovery controls and other important management aspects. Only by wrapping our management platform around the client infrastructure we can meet SLAs. We would know how the server is doing 24/7. Each server is like a patient in a hospital connected to sensors by multiple wires so nurses and doctors know what is going on. By deploying clients infrastructure on our management platform will allow clients to be certified by SAS70 or HIPPA or PCI relatively easy and painless.

We are patching servers at least once in a quarter. We feel that each server besides monitoring should be visited buy a human administrator at least once a month to just browse through the log. We are sending our techs to check physical alarms on devices every week. We operating backups to stay close or sometimes exceed requirements of SAS70 or HIPPA or PCI.

Besides basics we know that there will be daily client’s requests for changes, performance bottleneck resolutions, tuning, troubleshooting etc. For infrastructure of this size, there will be a hardware problem once in a quarter. There will be a need for security assessments on regular bases etc.

How can you include all that in the co-location cost?

Well, we were lucky. We got the information from the potential client and figured out what was wrong. The competitor gave them full management including EVERYTHING you can imaging in the IT world but limiting them to only 20 hours per month. It comes to 9 minutes per server per month!!!

If you take into consideration the excitement of the client’s executives that they can get FULL management for that cheap and our way of precisely assessing client infrastructure’s needs and quoting exact services and taking a risk of underestimating needed hours you would understand the disconnect.

Maybe we are just too precise to provide unparalleled consistency. Maybe we are stressing the importance too much?

Through latest sales cycles we can see that our competitors can go away with hazy definitions and still win the business. Get the client in the database canter then figure out how to make money on small things. This looks like novo days strategy of a lot of MSP providers.

I know one thing – the way Digital Edge treat’s its client’s technology – this is the key to the success. This is how IT groups of our clients become successful. This is why Digital Edge shows 100% uptime records for at least last 10 years. And this is why our clients say that their systems are stable and Digital Edge removes the stress from their operations.

Storage, thinking ahead…

Irene Goltsman — Mon, 13 Sep 2010 00:00:00

There is still a need for intelligence behind the technology even though hardware and software vendors are fighting hard to simplify complexity and hide details under the hood of GUI and wizards.

Coming from the system operation and integration business and sitting on sales sessions I see that all sales are done exactly like sales of electronic gadgets. “My hardware can do this, and this and this and even this!!!!!” At the same time if you look at your GPS gadget, you can see the percentage of the features loaded in the device that are actually being used.

The same logic should being applied when we are thinking about the storage. Before having the discussion about the storage, first companies need to access how it will be used.

Below are some simple questions that should be answered before interviewing storage vendors as each small feature will cost additional money and may be not utilized at all.

Do we need fiber, iSCIS, NFS/SIFS? Wrong answer to those questions may result in big additional dollars in licensing, hardware and software. If you are using iSCSI only then there is no need to purchase unified storage. If you need only high performance NFS, there is no need to purchase iSCSI and fiber. And it is not always true that you can gain max performance only out of fiber.
Disk speed. Disks are the most expensive component of any storage system. Same size standard SAS or SATA disk will cost 2, 3 or 4 times less expensive disk from EMC or NetApp or 3PAR. There has to be an intelligent analysis of I/O to purchase what you need. In a lot of cases vendors through in the mix thin provisioning, de-duplication and other techniques that also cost money. De-duplication would be total waist if you store compressed images. De-duplication and thin provisioning cost money as well and needs to be accurately analyzed before purchasing decision. Use of faster disks where it is not necessary is huge waste of money. There is confusion about I/O analysis. If there is a sequential reads then controller and switching speed is more important than disk speed. If you have short I/O, like database transactions, the importance of number if physical spindles (physical disks) much more important than anything else.
Do you need replication? And the most important is what kind of replication you need. There is a simple bit-to-bit replication or complicated mechanisms involving snapshots which will require additional disk space. The application needs must be assessed as we saw cases that our clients were purchasing complex replication mechanisms but application side could be using it. Also a lot of enterprise products have built in replication. Some enterprise storage replication mechanisms are priced per amount of data that needs to be replicated and add significant cost to the solution.