Affirmed Systems

10 Things a Start-up should look for in a Cloud Hosting provider

Mon, 13 May 2013 12:38:51 -0400

(Question originally posed on NY Fintech Start-ups meetup.com page - 05/12/2013)

Here is our short list

1. Where does the cloud provider host the systems? A good cloud provider should be happy to take you on a tour of the facility or at least provide you with documentation about the facilities, such as SAS-16 SOC certification.

2. Are the cloud virtual instances portable to another cloud? Does the cloud provider use openstack or cloudstack? vmware? Citrix Xen? Redhat KVM? How compliant is the cloud provider with other cloud based systems, such as hybrid cloud, private cloud, vmware, etc? What will the cloud provider support if the customer wants to migrate away at some future state? Can the customer take their cloud instances (virtual machines) out of the cloud easily to another provider or their own servers?

3. Explain all charges clearly! - What total charges exist? Does the cloud provider charge for incoming, outgoing or burst Internet usage or at a given speed? Avoid surprises - as at many clouds this can get costly!

4. API Support - What API’s does the cloud provider support to start-up, create, destroy, backup and clone cloud instances - how fast can a startup’s developer write code to have the cloud provider’s system easily scale via dynamic methods on demand?

5. SSD and Flash Storage - Does the cloud provider use extremely fast SSD storage (greater than 10k storage operations per second etc)? If so for what price? Many cloud providers use extremely slow disk - on purpose - to force customers to purchase virtual instances with more RAM than necessary - this is very expensive. At Amazon an instance with 63GB of memory is far more expensive than a smaller one with 7GB of memory - but the disk is so slow the developers often have no choice - they need to rely on ram for databases and such. A start-up can get caught in the “memory trap” where in order to get more performance from their site or applications the start-up will have to spend more and more money on renting memory from a cloud provider to overcome slow disks…

6. Performance metrics the Cloud Provider will publish - What is the published performance and other customizations for performance the cloud provider has made for load balancers, NoSQL databases? How fast is their high performance computing platform (HPC) from a storage and network layer - many start-ups are now building systems where a message bus is critical or complex event processing (CEP) is part of the system. An example is how well can a cloud provider’s systems run Streambase, Tibco RV or use GPU’s for big data?

7. Support and SLA - Does your cloud come with service? How well supported is the cloud provider’s systems? Can the start-up pickup a phone or open a ticket and quickly speak to an engineer in real-time? If the startup does not have Infrastructure Cloud experts on staff (like Adrian Cockcroft of Netflix) - the startup may not be able to fully understand all the issues they are facing during an outage of their software and may not be able to adequately configure their applications for fault tolerance.

8. Cloud Security - Can your cloud provider provide a secure isolated environment for financial transactions and PCI compliance? This is critical from a payment card and financial trading perspective. If your site is tested by your merchant processor and fails you could be in for fines and hefty charges to re-architect your site to comply.

9. Private connectivity to their cloud - Can your Cloud provider allow private connectivity to the cloud based systems? An example would be a trading firm startup wanting to use cloud, but needing fiber optic cross connects to a broker, bank or ECN for live pricing and trade execution. Some cloud providers can incorporate direct secure access into the cloud systems.

10. Cloud Disaster Recovery - Does the cloud provider support snapshot or backup services to a different geographic area of the country or world? This may be critical to the start-up to provide their own customers assurances and meet RFP questions when the start-up is leveraging their cloud provider’s infrastructure to win business. Of course the next part of this is HOW FAST can a start-up recover from an outage. Ask your cloud provider to provide detailed reports of how fast virtual machines can be restored from backup to live machines.

Thanks,

Joe Brunner
CCIE #19366
Affirmed Systems CLOUD ASSURE™

2012 in review: The "Living backup" is now the IT standard

Thu, 20 Dec 2012 07:02:00 -0500

Affirmed Systems is a real-time managed services provider that monitors, detects and alerts on customer performance metrics 24 hours a day, 7 days a week.

The history of backups

Data backups for many years have been static, or monolithic copies of production systems taken at scheduled intervals. Often backups were stored on magnetic tape for easy offsite archival, or on dedicated appliances. In some more advanced environments, storage area network (SAN) systems perform backups between 1 or more SAN disk arrays, either within a single site or between multiple locations to achieve disaster recovery and data protection.

These backups did protect the organization from complete loss of data in the event of disaster such as fire or flood. However these backups were NOT very good at avoiding downtime due to less permanent disasters - power or connectivity outages, server crashes, raid card failures - all things that routinely took production systems offline and effected business operations.

Redundancy solutions or products that complement a backup

In addition to doing backups, many firms deployed products like DoubleTake and could replicate live systems to hot or warm standby systems. Redundancy solutions native to the operating system or application became mature - such as Exchange 2010 Database Availability Groups and Windows SQL Clustering. Coupled with global server load balancing technologies and dns change mechanisms - the “server farm” could be highly redundant and available even in the event a complete site went offline. The challenge of these products was IT staff would have to train and document procedures for bringing the disaster recovery site online and synchronized to run as the production environment. The complexity and costs of building and maintaining these redundancies also kept their increased up-time gains out of the reach of most small to medium sized businesses.

Combining backups with redundancy to create the “Living Backup”

Our Managed Services customers often ask us when we integrate a new or existing environment in our platform - “What type of backups do you do?”

Our answer to the customer is “Living Backups”. A Living Backup is backing up a server or filesystem as a replica that is deployed directly into a virtual cloud platform. The backup is a live, up to date copy of the entire system being backed up. At each interval, the backups re-seed and update the replicas. This provides the customer a ready to go, warm standby system with optional geographic diversity from their primary server facility or office.

The backup is now no longer static, or monolithic but completely useable in a self-contained format offsite. Software products from Acronis, Appassure and others allow production physical or virtual servers including all the live data to be replicated off-site into a private or public cloud, ran as live servers, then replicated back easily when the primary site or server is restored. This technology is affordable and can be setup for a small fraction of what this type of redundancy would have cost 5 to 7 years ago in DoubleTake or using a SAN to facilitate data replication.

The Living Backup strategy is also very secure and through version control, 2 to many revisions of each snapshot backup can be kept available in the cloud or moved to other online or offline backup media facilities by the Managed Services Provider.

Current trends and how to capitalize on them

Private and public clouds for running Living Backups are now affordable and within reach. Many small businesses run software and database systems that must run on extremely fast, high performance physical servers, with SSD raid or Fusion-IO based storage systems. These systems are proprietary, generate tons of revenue and can not use public cloud services such as Amazon, Rackspace, Google apps etc. These systems must also have tremendous levels of backup and resiliency - they will need to be available at all times and under all conditions - regardless of hardware or network failures.

In 2012 we converted our last few tape backup customers to fully redundant offsite virtual systems on our private cloud. Our standard customer production deployment is vmware esxi on the bare metal with Windows or Linux guest servers. After testing many backup products for vmware esxi, we standardized on Acronis vmProtect 8. This product utilizes Changed Block Tracking (CBT) technology to backup changes using virtual disk file delta’s. After an initial full backup, only the virtual disk delta’s are then backed up to our cloud. Acronis vmprotect “updates” the replica using a delta merge process to update the replica’s virtual disk, which contains the entire operating system and all production data. This process happens without effecting the production servers as frequent as every 30 minutes. We tested even more frequent replication, but so far 30 minutes has been a good fit for our customers. For customers who do not yet use vmware in their production environment, we utilize Appassure for doing Living Backups of Windows servers. For Linux, we use proprietary internal scripts and packages to keep replicas continuously updated from production to our cloud.

Lessons learned from Hurricane Sandy

Hurricane Sandy struck New York City and took dozens of public data centers offline in Manhattan and the surrounding area for days. Companies large and small as well as government agency server farms, many inside offices were completely offline due to power and Internet outages. Many data center facilities that had backup generators found issues with their generators once running on them and lost power multiple times. This took systems offline repeatedly, corrupted data and damaged components. It was a painful and expensive lesson many firms are still struggling to recover from.

During the storm and immediately after, our team swung our Living Backups into production - by changing dns records of publicly available customer systems to our cloud. For customers working from home, internal systems were available at our facilities via backup VPN and remote desktop services. Some of our customers were up moments after Sandy disrupted power service in Manhattan, already online in our remote data center locations. Some customers required special procedures we had not planned for unique to their organization. We worked around the clock to restore production services - starting even before the storm had left the area. In the coming months, our procedures will become even more defined, adding more standardization and policies to prepare in advance for supporting proprietary as well as common systems. While there is often no “one size fits all” approach to Cloud backup and availability - testing and reviewing backup reports multiple times per hour and per day insures customer data is always available where and when it needs to be.

Thank you to all our customers for making 2012 a great year. As always, its a responsibility and honor we live by to make sure your systems and data are safe and secure.

Happy Holidays!

Affirmed Systems

www.affirmedsystems.com

Why we dont do Dell - A horror story from a channel partner...

Tue, 21 Aug 2012 22:13:24 -0400

We worked with the Channel sales team out of Austin, TX last year… we ordered six figures of dell servers, san and MS licensing… we received the wrong san and our client was livid - they only had the week of 10/31/2011 to get a migration done and ordering the equipment 9/18 we were told no problem to have it delivered by the second week of October - very stock environment - nothing specialized… the wrong san arrived and Dell would not replace it unless my company - a channel partner - wired funds for the the replacement SAN… This put my company at great disadvantage - in desperation we reached higher and higher up, finally reaching the head of channel sales using linkedin to find him - the best our dell team would do is have us ship back the wrong san, issue a refund check and then we would have to buy the right san all over again - this process (the RMA alone took 10 days to get issued after we returned the wrong SAN). Dell’s advise to me? just wire us more funds - basically - they would do nothing for us UNLESS we obviously made it look like my Dell Channel team could “ring up another sale”… very sad… the client eventually missed their window and fired my company as their IT integrator and datacenter hosting firm citing our performance on this cut over as grounds for termination. they cancelled a 3 year managed service and hosting contract - worth MID-SIX figures to my company… So in short, I would not never use Dell again for anything - not even a workstation. IBM some how found us and reached out to us - Since switching to IBM its been a world of difference… their Channel Team has local people who CAME to our office and trained us on their product lines for 3 full days… we have received EXCELLENT support from IBM all along and they are a bit less expensive than Dell (its a myth they are more expensive). Finally also one more thing I have to mention with Dell - when they buy a company - in this case Compellant - the Dell and Compellent teams are not well integrated. We brought another big client to the Dell offices to meet with Dell and Compellant - they could not give us a quote and the pre-sales engineer they assigned only had instructions for taking logs from vmware ESX (not ESXi as is widely used now)… I had to explain to the engineer many times that ESXi does not have a Samba interface so they log dump to a path would not work - i modified the process using my own ESXi knowledge - they stopped responding to emails - all the while Dell’s channel team was not managing this process and the client got very cold feet about dealing with Dell/Compellant… he instead purchased a SAN from EMC through an EMC channel partner - costing us the managed SAN contract and sale we would have got had Dell/Compellant come through… I could go on and on with Examples - but I would avoid Dell at all costs - its clear they want to be come a software company and their channel is not what I expected at all.
Joseph Brunner
CEO
Affirmed Systems

12 questions any company should ask a managed datacenter provider

Wed, 15 Aug 2012 16:39:00 -0400

Affirmed Systems is a managed data center provider that specializes in hosting financial trading and real-time sensitive customers at Equinix, including disaster recovery facilities for customers with primary data centers currently at in-office facilities.

Recently, we spoke with a start-up firm that was considering quotes from several other data center providers. After reviewing these quotes and doing some research we noticed the offerings varied greatly. To help clarify what options exist in the market place here are 12 things you want to nail down before choosing your managed datacenter hosting provider.

1. Does the provider own & operate their own datacenter facilities or is the facility a leased space in a larger facility?

Managed datacenter providers who do not own their own facilities often cater to customers looking for 1 to 5 racks of datacenter space. This may be a benefit to smaller customers IF the facility exists in a larger facility such as Equinix or Savvis - the customer would be gaining power, environmental controls and security monitoring by leveraging their datacenter managed service provider’s host facility, without signing a contract and leasing space from the facility directly. Be wary of any managed datacenter providers that will not identify who owns the space their facility is hosted in or if they will not provide a onsite tour PRIOR to signing a contract. Such facilities are often called “Lights out datacenters” - make sure the “lights wont be out” if the facility is too small or is not well powered or exists in a converted office space. These facilities may not have power generators, industrial grade cooling and reliable fire suppression larger facilities have.

2. What Internet and Telecom carrier options exist in the facility? Is the provider’s facility “carrier neutral”?

Being carrier neutral is important to companies that have an MPLS or Wide Area Network they will want to connect to the datacenter facility where their systems are hosted. If the datacenter is not located in a central “carrier hotel” like 111 8TH Avenue in NYC or 350 Main Street in San Francisco, the only option may be third party carriers and higher than expected per megabit charges. Recently while putting together a quote for a customer that required datacenter facilities 25 miles outside of their primary location - we found per megabit costs were MUCH higher than expected. It turned out the only option in that datacenter was to use the datacenter owner’s internet bandwidth network at a cost 4.5x higher than the per megabit costs in other datacenters we host customers in. Additionally, carrier neutral facilities allow the customer to receive managed datacenter space from one provider, but bid out bandwidth and private line services to many carriers, forcing them to compete for your firm’s business. Sending a single quote request out to 10 or more carriers allows your company to receive all options and choose the carrier that is the best value for your business.

3. What security restrictions and other physical security measures are in place at the facility?

Even now this is often overlooked at many managed datacenter providers - access badges that lack biometric or user specific access controls often allow misused or lost access badges to put access to the facility at risk. Also be sure to ask how is access to my equipment monitored in a shared area? Are customers of the datacenter provider working in a shared area, where other customer equipment is hosted? This could be a potential accident or malicious event waiting to happen. Many servers look alike, mistakes happen - your up time could be at risk if security is overlooked either from mistakes or other issues.

4. What shipping and receiving restriction or costs is my firm subject to using your datacenter facility?

How Difficult will it be to ship and receive packages from the facility? As a potential customer with a budget, you should know if there will be per incident charges or per package charges associated with returns, part replacements and general equipment servicing. One datacenter provider we reviewed a quote from is charging $200 PER HOUR for any time the managed datacenter provider spent boxing, shipping or scheduling any equipment deliveries… This can easily add up to THOUSANDS of dollars if servers or equipment fails. While the datacenter provider has to remain profitable - make sure you do not pay for any poorly negotiated contracts your managed datacenter provider entered into with their own host facility.

5. Who is responsible for on-site service calls? What is their time to respond to the facility?

Many datacenter hosting providers have talented, skilled engineers on-site 24 x 7 x 365 to respond to customer system alerts - however many rely on third parties, or the on-site staff of the hosting facility they are leasing space from. Slow response times can easily cause a mistake or minor outage to last for many hours. Tripped power circuits, failed power supplies, misapplied IP addresses - have all caused customers hours of unplanned downtime. Don’t be a victim of a “ghost town” datacenter - make sure you have a clearly defined SLA of how soon an engineer can be expected at the facility in the event of equipment failure. Get a detailed list of first level and management escalation contacts for your managed datacenter hosting provider.

6. If your managed datacenter provider provides data backup solutions - who monitors those backups? Where are the backups kept?

Often managed datacenter providers bundle data backup services in their standard service offering. It’s essential critical data is kept offsite -even if the primary place it’s hosted is a datacenter! Fires, natural disasters, other events happen. Your managed datacenter provider must identify and protect its customers (YOU) from risks of data loss. Also another important thing to ask about is what additional charges are associated with using the managed datacenter provider’s backup service. Will there be per Megabyte, Gigabyte or Terabyte transfer costs passed on to you? Will there be latency or bandwidth delays associated with backing up your data? How fast can the managed datacenter provider backup or restore your data? What is the expected transfer rate in Megabits or Gigabits per second of their backup systems to the offsite facility? DO NOT assume its fast enough until you ask! At one facility, we observed the customer’s “gigabit” offsite backup service running at 40Mbps (~T3 Speeds) and unable to complete the backup in a single evening - this is important as new data was being written FASTER than the data could be backed up!! This made the service useless and another more costly service had to be provisioned… of course at expense to the customer!

7. What time frame does the managed datacenter provider have to respond to upgrade and general support tickets?

A recently reviewed contract was very sparse on details regarding support time frames… We recommended the customer push further for more details. The customer was not able to ascertain what amount of time provisioning a new virtual server or upgrading the memory in an existing virtual server would require. Uncertainty of service time frames can lead to unexpected delays to your company if not addressed prior to contracting managed datacenter services. Providers are looking to grow their revenue while minimizing their own costs. Make sure any contract you sign meets or exceeds your own service requirements.

8. Does the managed datacenter provider have a 24 hour support hotline?

Key to gaining timely support is being able to speak with a live engineer who can address and often resolve or quickly escalate issues - especially in an emergency. Time is money and your firm can’t always resolve critical issues from an online ticket portal. This is where it pays to ask for a few customer referrals and for the number you will be calling to gain support. Call the number at 3AM on a Sunday morning. See what support from any potential providers will be like BEFORE signing the contract.

9. Does the managed datacenter provider offer leased hardware, virtual server capacity and spare parts service?

Your firm may need equipment in a hurry. A project may need more compute and memory than your own companies servers permit. Does the managed datacenter provider have the capacity to provision additional secure systems for your firm to utilize? What if your company’s management wants to run a demo of an accounting application they are considering, but does not necessarily want to go out and purchase more servers to do it?

Can your managed datacenter provider offer a few virtual machines for a month or two (or longer)? How quickly can this be done? Also, given delays and risks to production - Can your managed datacenter provider QUICKLY replace failed power supplies, hard drives, memory, fans and other common devices? Does the managed datacenter provider offer parts replacement services?

10. Does the Managed Datacenter provider offer Advanced engineering and complex equipment configuration?

One of our customers came to us looking for a managed datacenter provider that could offer engineering and technology support personnel that could install and customize load balancer’s, access gateways and manage their MPLS network. Does your firm require ad-hoc or additional services from your managed datacenter hosting firm? What costs are associated with these services? Are charges per item? per hour? per month? If your firm requires these services choose the managed datacenter provider that can step up to the plate when your own IT team needs assistance!

11. Does the managed datacenter provider offer server and equipment moving?

Many managed datacenter providers will only rack and install the equipment you ship to their facility. Does your firm require assistance and planning of a complex move, considering logistics, downtime and post-move application testing? Ask the managed datacenter providers you are considering what is required of you to bring or ship equipment to their facility and what other options exist? Some managed datacenter providers specialize in moving from an existing in-office datacenter or other hosting facility and delivering your equipment to their facility.

12. Does the managed datacenter provider offer a Shared or Dedicated Network and Firewall infrastructure?

Let’s face it - your systems must be secure at all times no matter what. Accidents and surprises are NOT acceptable. Does a shared network infrastructure pass your own internal IT security policies? Many managed datacenter providers are looking to re-sell capacity on existing network switches, firewalls and routers they already own and service. This allows them to maximize returns from both their own capital outlays and the limit amount of customer equipment they have to host in their own facilities.

Do not accept the promise of segmentation or firewall controls that will protect your firm’s systems from another customer’s. Get the details on how your security will be assured and managed. What Control procedures exist on the provider’s equipment to alert and track changes to shared equipment? A single line can be accidentally changed in a shared firewall’s configuration and your own systems could be unprotected from the internet or any compromises of another customer’s managed systems. This came up last week when a prospective customer was told they would not be allowed to utilize their existing Cisco ASA firewalls in the managed datacenter provider’s facility. The provider wanted to sell them a Cisco virtual firewall context on a shared device used by all customer’s. This may or may not work for your firm…

Please review our website for managed datacenter hosting options and ideas.

Joe Brunner, CEO

CCIE #19366

Affirmed Systems

Does a "Shared Nothing Architecture" make vmware's vmotion high availability unnecessary?

Mon, 23 Jul 2012 21:48:00 -0400

This month we are in the middle of a data center consolidation and architecture clean up for a new customer that has “gotten by” on vmware free edition, open source distributions of Linux at their Data Center and a few Amazon EC2 instances in the cloud for some time…

When putting together the budget for their technology roadmap, we stopped to reassess what version of vmware we needed to purchase. We are big fans of having vcenter to do hardware alerts, virtual guest and other monitoring natively, so we definitely need a paid license of vmware. We always recommend purchasing an Essentials or Accelerator “kit” which is a bundled package of vcenter server, usually allowing 6 physical CPU’s and 32GB to 96GB per CPU vRam entitlement in vmware vsphere ESXi 5. In our vmware deployments our license (and budget) decision is based on these questions:

1. Does the customer have a Storage Area Network (SAN)? Will they be purchasing one? Does the budget exist for one? Do they need one?

2. Does the customer need vmotion? Will there be a single virtual server that needs high availability (HA) from vmware? Can we do HA via Exchange or SQL server clustering (less expensive than doing it using vmotion and a SAN)?

3. Does the customer need other vmware “enterprise” kit features such as distributed virtual switches and Distributed Resource Scheduler (DRS)?

4. Can the customer make do with a simple, affordable vmware essentials license for 3 or so servers (every 6 physical CPU’s) if only plan to deploy 3 servers each with 2 cpu’s in vmware now?

This link from vmware’s site is a quick reference on the different essential and accelerator kits available -

http://www.vmware.com/products/datacenter-virtualization/vsphere/small-business/compare-kits.html

While our initial instinct was to purchase a vmware Accelerator Enterprise kit (about $20,000), we evaluated how a “shared nothing architecture” will effect the customer’s needs for single virtual guest server high availability. A shared nothing architecture is a system where every server has its own copy of the data and replicates add’s/delete’s/updates/changes to one or more nodes, such as the data nodes in a MySQL cluster.

Facts and observations:

1. The customer does not currently own a Storage Area Network Device (SAN), so even with a high end vmware Accelerator Enterprise kit, their applications can’t survive the failure of a physical server. Without a centralized SAN, applications and databases need to move to a “share nothing” model, where entire copies of the applications and databases are clustered or replicated to mulitple guest servers, each running on different physical vmware hosts.

2. Does the customer need single virtual guest server high availability if their database is going to be running on a MYSQL 7.2 Cluster with each database replicated to multiple data nodes, all with their own online copy of the data?

3. The cost for a decent SAN that can store 5 TB now and 10 TB within 24 months and a vmware Accelerator Enterprise kit license would run about $90,000. This does not include the annual recurring renewal costs for a SAN warranty and vmware software upgrade support fees (about $15,000 per year for both).

4. The customer has legacy Intel based servers, 2 to 4 years old with very basic drives and performance crunches effecting the business. We feel their budget would be best spent on some very very fast servers using high capacity eMLC solid state drives in a raid-5 or raid-10 and Intel E5-2670 processors to increase performance.

Designing the customer’s high availability, we took into account how well database clustering works now with MySQL 7.2, jdbc or ado.net connectors in the applications pointed to 2 different database server targets or a single target comprised of multiple servers in a load balancer pool. Its hard to justify spending dollars to build out the vmware/san model if its not needed to gain high availability.

For most of our customers, vmotion and Storage Area Network sans are really best used in a “shared disk” environment, where a single virtual guest with a single copy of the data “MUST BE ONLINE AT ALL TIMES”. This is common design, where legacy applications can not be easily replicated or ran on clustered database servers, etc.

After careful consideration, we decided to recommend only the vmware Essentials license for the budget ($600) and use the $19,400 or so we saved on a vmware Acclerator Enterprise kit towards some really well built out servers that will make much more of an impact on the customer’s application performance, while providing high availability.

In summary, high availability of applications and databases can be provided by a distributed, shared nothing architecture in software, allowing the failure of one or more cluster member servers without effecting production. This design allows CTO’s, system architects and technology integrators to re-evaluate whether vmware based high availability is required, which can lead to a huge savings in the enterprise!

What is a Managed Services Provider?

Tue, 10 Apr 2012 15:31:00 -0400

There seems to be a lot of confusion about what a Managed Services Provider (MSP) is! Some people think it’s a vendor manager. Some people think its a group that runs software agents on the workstations and servers at customer’s offices. Still others, think MSP’s continually backup all the Company’s data to “the cloud” and can quickly and painlessly recover any file, spreadsheet, user account or any other valuable object someone deleted.

In fact it’s all these things and more. Its certainly proactive and a clean break from “IT Consulting” which conjures up images of a fat guy showing up with a backpack and Starbucks cup when someone can’t print. (no offense to fat guys or Starbucks).

So to help educate and inform here is Affirmed System’s list of “MSP Must’s”

1. Managed Service Providers MUST monitor their customers networks, servers, phone systems, workstations, firewalls, routers AT ALL HOURS. Even 5 minutes to midnight New Years’s eve or at 10pm on the 4th of July during the fireworks SOMEONE must be on call to receive alerts, or be sitting at “big screens” getting alerts in real-time. Companies live and die these days on productivity - and that is why they hired you Mr. or Ms. MSP. If they wanted to call an IT guy when they realized their network was down they wouldn’t pay you every month to protect them from that!

2. Managed Services Providers must fully backup and protect all customer “assets”. Servers & workstation OS images, files, databases, network configuration files and have a clearly defined daily process to insure MULTIPLE people are responsible reviewing backup progress and success. There will be a spreadsheet someone in Accounting CRTL+SHIFT+DELETED and YOU will be asked to recovery it quickly from the most recent backup. Just as important to daily backups - you should plan and test total a systems recovery at an offsite physical or cloud based infrastructure. Even if you just keep 10 Amazon EC2 instances on standby for your team to test recoveries to - the experience and knowledge to do this again, quickly, is what makes you a managed services provider. Figuring out your total disaster recovery strategy the day you customer’s server room is flooded or had a fire will not help them get back online quickly…

3. Managed Services Providers must keep an effective inventory and security policy in place to insure all customer assets are well tracked, warranty statuses are updated and phone numbers are handy for hardware and software assurance contacts. If you have a server warranty, you need to have a central portal or repository to find Service Tag’s or Serial numbers, out of band management card IP Addresses and login credentials, phone numbers to call to gain hardware assurance. Time and time again you will be tested when something fails, so keep all this information handy. Waiting until “all the server’s lights are red” is a bad time to start googleing “HP Business Support 800 Number”.

4. Managed Services Providers Must have an effective, easy to reach help desk that is staffed with qualified experts who can actually resolve issues. An answering service IS NOT A HELP DESK. I’ll say it again, because there seems to be some who think otherwise - “AN ANSWERING SERVICE IS NOT A HELP DESK”. Having your frustrated, valuable customers call into even a pleasant, well rehearsed offshore national or recording service is NOT an effective way to ask customers to gain support. This will only show you have HUGE gaps in your coverage and are not “on the field with a full team”. Many MSP’s literally walk out of the office and forward their desk phones to an answering service. Truth be told - our sales team will eat your lunch on this fact alone every day of the week. Some of our best and most loyal customers have come over to our MSP program for this reason alone. Be warned.

5. Managed Services providers must listen. Its not enough to just automate IT and standardize IT. You’ll be need to do that anyway just so you can keep up with all the proactive issues you monitoring is finding and have a good framework that serves all customers equally. You must also be responsive and timely in handling the basics and the not so basic requests that make each of your customers unique. The best managed service providers can slip in a powerful framework while still maintaining trust and the absolute guarantee - you are not just “some IT Company” but THEIR IT company. And that’s why they work with you.

6. Managed Services providers must innovate. Its not enough to copy what’s going around in the industry. Or load canned alerts into your MSP Platform and hope they’ll catch viruses, smart HDD alarms, raid-card failures and such. You need to dig deeper and find the fresh approach THAT THE PREVIOUS IT FIRM AT YOUR CUSTOMER DID NOT. You may not find all their issues during your “integration survey” or even on your “MSP services roll out”. But through constantly being aware of what great new technology options benefit YOUR customers you will be greater than the sum of your parts. You will find ways to continually improve as you innovate. The reason for innovation…

Thank you,

Joe Brunner

Affirmed Systems

Are you getting the performance from your network you deserve?

Mon, 09 Apr 2012 11:07:54 -0400

Networks can be both a cash cow to companies that optimize them or a tar pit of wasted time to companies that do not understand why things are “slow” or “unresponsive”.

Affirmed Systems is a managed services and Data center hosting firm that is the industry leader at analyzing network performance metrics and dash boarding out how your network is working.

Here are some of key areas our team will identify:

1. Internet access and topological availability - where are your services fast from? Where are they not fast from? Do you have users abroad that need to reach services hosted at 277 milliseconds away from them and get very little throughput? Are your offshore developers waiting too long for screen refreshes on Citrix, Microsoft RDS or other real-time desktop technologies? How fast and responsive are our internet circuits at baseline, busy and top hour periods? Where are our slow downs? Who receives and acts on bandwidth alerts within a time frame that is helpful to addressing risks to user’s performance?

2. How responsive and flexible is our converged network to real-time voip and video? Do we have the proper network equipment and configuration to host video conferences and voip conference calls that will look and sound great? Are our Polycom and Cisco network video based systems waiting behind emails or other non-realtime sensitive flows? How resilient is our converged network to single points of failure, that can cause our converged network to suffer downtime and sluggishness?

3. What applications are performing well and where can we improve? Affirmed Systems can quickly identify and help you improve overall bandwidth utilization and requirements from the application layer for local and cloud hosted applications. Are our users getting the best response time in-house or when working remotely? Do we have the “right” type of bandwidth available at each facility or Data center? May firms “buy into a well established name” Like ATT or Internap and do not realize that these firms have their own unique networks that work well in some areas and not so great at others.

4. What is my high availability score? How redundant and responsive is my network to single or multiple carrier issues? How fast does my network detect outages and self-heal around them? Do I need A CDN? BGP? High availability DNS? How well connected is my network, did my internal IT staff overlook anything? What could we gain by making changes? Affirmed Systems “knows before you go”, using our best practices proved in many industries, such as retail web site management, financial trading, service provider and data aggregation networks. On a scale of 1 to 10 we will present you with your “High Availability Report Card” in an easy to understand format - this will make sure all IT stake holders are up to speed and able to present to senior management areas where the Company should invest it’s resources to improve.

Contact Affirmed Systems today to get started.

sales@affirmedsystems.com

1 (866) 973-9933 x3

Top 10 things a great Data Center Manager will do

Mon, 19 Mar 2012 03:19:00 -0400

What does it take to be a Data Center manager with real-time responsibilities, supporting the hardware, systems, software and network performance critical to an organization?

I thought I would share the list of top 10 things we are focused on in our Data Center management team and to hear some more great ideas and feedback from other Data Center managers!

1. Use multiple systems to monitor. No single monitoring system no matter how well built can possibly be responsible for insuring up-time. Too often Data Center managers trust a single platform or vendor to completely manage disparate systems such as switch infrastructures, edge routers, virtualization platforms, etc. Despite all the sales and marketing lately from the industry about “Data Center 2.0” being a “single fabric” or “end to end platform” - there are still crucial metrics and key events that will not be tracked on a single platform. How full is my storage? How many multicast routes do I have from a particular trading exchange? How many firewall rules exist for a specific application? The list goes on and on. While its wise to have different software based systems “cross monitoring” from both internal and external vantage points - internal monitoring systems such as embedded controller cards should not be overlooked. Alert agents that run directly inside the systems themselves can often provide unique insight into system health and can work autonomously of external snmp, rmon or agent polling. The key thing bringing all these different systems together is a solid business workflow, where the Data Center staff are aware of what monitoring systems are responsible for what and have a well defined remediation policy in place to know “where to look” during an alert. Only by fully understanding the capabilities of each platform can your team be certain to be alerted when an outage or risk exists. Finally, do not forget a “pre-flight” check - at each shift change or at least right before the start of the work day. This is a comprehensive deep dive into the performance and state of all systems and services that are critical.

2. Cross train your teams. “It must be the network” (sound familiar). Too few Data Center Managers take the time to properly cross train different teams. The team managing the vSphere cluster should understand vlan tagging and how 802.1q switch trunks work. The Network team should understand how a vmotion is done on different network and storage systems. Your database administrators should have the list of IP endpoints the SQL Servers and clustered systems listen on and interconnect with. When something is down having a well rounded team is ESSENTIAL to quickly identifying the issue. Frequently, a Data Center Manager becomes a “translator” between different groups, that often blame each other during a failed deployment or real-world outage. By making sure each team has enough knowledge to “cover the gaps”, the Data Center teams can rapidly pin-point and resolve issues.

3. Have Logical, Physical and Wiring diagrams handy. Keep them up to date. In the era of partial or complete process outsourcing, having quick access to how the systems and network topology is cabled is just as important as understanding packet flow and firewall access. You may need to bring a vendor who is responsible for (but has never been to your facility) up to speed quickly before troubleshooting can start. The Diagram is often their only insight into what they are supporting. Successful Data Center Managers become “Librarians” of documentation about their facility - and the best ones have it available via a cloud based portal or other offsite, yet easy to access secure system. Its going to be hard to pull your Visio’s off of SharePoint if the core switches are down, huh? Google Apps is a secure, painless and often free way to insure each member of the team can quickly reference any document in the technology organization. To keep documentation updated and accurate, assign “Update Managers” who are responsible for verifying and checking off on documentation updates weekly, monthly or however often your facility schedules changes.

4. Take responsibility for outages. No one likes a mystery. Especially one that takes a business down. If you are aware of what caused the outage - document the problem thoroughly and inform all stake holders without blaming anyone (even if someone should be blamed). The truth will always come out anyway, so its better to be truthful and let everyone in the technology team learn from the mistake. Do not blame Microsoft, Cisco or some vendor for an outage. These vendors make ALL necessary documentation and support personnel available and you will not be taken seriously if your outage report says “Microsoft Dropped the ball on this one”. A year ago, a major Microsoft Exchange outsourcing vendor sent us and our customer’s Executive team an outage report - they simply stated that after speaking with Microsoft it was clear their Exchange Mailbox Cluster Servers needed some patches they didn’t have that caused the cluster to fail back and forth frequently between members of the mailbox cluster. Microsoft was not at fault and the vendor was seen as more responsive to a serious business day email outage for having taken ownership of the fact they didn’t have their systems patched properly - rather than blaming Microsoft.

5. Live and Breathe your backups. Data is the priceless human capital that above all you are tasked with protecting. Data is your “VIP” and the Data Center Manager is the head of its security detail. No backup should be considered complete until a RESTORE test has been done, preferably offsite to a different set of systems. A great Data Center Manager will assign different team members daily tasks to insure backups are accurate, complete and can be used to restore any file, system, database or anything else that exists under your control to its working state. Do not simply trust that an outsourced backup provider is “doing your backups”. Demand proof, in the form of both a login to their system (any good outsourced backup provider will have this available) and at least a quarterly restore of both the entire systems and the data that live on those systems. Many organizations hire a specialist firm to do their backups, yet have never done a restore test? Why not? This should included for free as part of their service, if not shop around - your firm deserves better! If you do your backups in house, assign different team members to cross check each others backup tasks. Some organizations have a Database Backup policy or process that is different than how they backup and archive email. Do not allow “backup completed successfully” emails to guarantee both backups and restores will actually work. Too often the only thing that worked was that email.

6. Make email alerts informative and useful. Too often we see alerts that simply say “line down, interface X/Y/Z” or “system alert - 95% disk utilization on Server A”. An engineer now needs to track down what is wrong and spend valuable time while something is down, determining the root cause of the issue or who should be contacted to resolve it. Your alerts should clearly state as much information as necessary to allow the groups responsible to take action immediately. Wouldn’t this be alert much more useful (especially on a Saturday Night when the Network Engineer on call is at Applebee’s with his family)?

Router Alert: Edge1-DC1-US.company, Interface G0/0 - Down - Circuit ID 10-9A89AB30 to Cogent down, Contact number 1-877-726-4368, option 3, 1, 2

7. Train your team on what to do during an outage. The first time something is down is a bad time to find out who the best Exchange Mail store troubleshooter or SQL Cluster Expert in an organization is? Or who knows Multicast PIM works differently between the different data feed vendors the company relies on? For each potential “hot spot” in your Data Center (anything that if down or slow someone will notice) - a well defined remediation plan should be drilled on - “What interfaces does our switch receive multicast data on, what interfaces does it send multicast data on?” “What vswitches have and use jumbo frames, and which SAN interfaces on which switch ports need jumbo frames enabled”? “What Luns needs to be available to which vmware host for Exchange or SQL to be available”? “How does a vmware host look or respond if it can reach the storage, how do we troubleshoot vmware nic connectivity issues at both the host and guest levels?”. A well trained team is a pleasure to work with. A poorly trained team is obvious and won’t go unnoticed, especially when each minute of downtime is costing the company money and credibility.

8. Know your min, max and average power drain. Sudden changes in power utilization can be an indicator your Data Center power feeds have become unstable, or the temperature is unbalanced in hot and cold aisles, etc. If a server has dual power supplies, each homed to an “A” and “B” PDU in your cabinets, guess what happens if server’s power supply has failed? - it will drain additional power it needs from the single remaining power supply, causing more AMPs to be drained on a single PDU. Your team should monitor and alert on each PDU and if possible the entire aggregate of all PDU’s in your facility, using a system such as APC’s InfraStruXure system, or by doing a sum function of power “Amperes in use” across many PDU’s on a single graph. Work with your power provider to document total available power and how power is distributed in your facility. Create power alerts that indicate how well the Data Center is functioning.

9. Familiarize staff with the physical layout of the Data Center. No one wants to trace cables or work through a nest of SAN cables the day a disk shelf is unavailable or system is down! Who wants to try and hear what someone is yelling to them on the phone in a loud Data Center! (Assuming you have cell phone coverage there at all) Take the time to show each member of your team where fiber, cat5/6, telco lines come in to the facility. Which patch panel numbers go with the top of which cabinets if you use a centralized wiring scheme? Is “12-A-11A” rack “12” or row “12”? Which ToR (top of rack) switch or fabric extender runs back to which core switch and how are the uplinks labeled? Nothing beats experience under fire, and no one can outsmart familiarity with an environment when the minutes tick by during an outage. We call them “Data Center Parties” and we have them often! They are an opportunity without the stress of something down for each person to see how things connect and how cables are labeled, know what all the lights mean, appreciate how something is setup that you may not have had knowledge of. Great Data Center managers all have a role in keeping the team familiar with how the facility is laid out.

10. Have a vendor escalation plan. Janet may be your sales rep at your SAN vendor that gets back to you quickly when you need a quote. Paul may be her manager that sent you that huge tin of caramel popcorn last December. Do NOT expect them to answer their cell phone at 3AM if your SAN is down. Each vendor for telco, hardware, software or process outsourcing should provide an easy to read, effective “Escalation Contact Sheet” that insures you and your team can reach whoever you need to speak with to restore service at any hour, even on a weekend or holiday. Do not assume opening a support ticket with the 800 number of your vendor guarantees you will get help that fixes the problem in a timely fashion. Many well known vendors use an answering service for late night calls, that act as more of a call center than an engineering point of contact. You will then need to wait for an engineer to be available or for them to do research on your issue before getting a call back. It can be many hours before “someone who knows” is actually working your issue. If something is critical to the success of your Data Center’s operation - have second and third level 24 hour contacts available. A few years back, a large Telecom company gave us an Escalation Contact Sheet with 24 hour numbers for staff there as high as two people below the CEO with home and cell phone numbers as well as his peers and people in his office. While we don’t recommend calling them first - clearly use your discretion and best judgment - if your ticket has been sitting idle in a queue for a couple hours and you’re getting nowhere - your management will expect you to be able to escalate the issue - so be ready and able to do so!

Thanks for reading and we would appreciate some feedback or additional areas we should be focused on!

More cases of ISP google Hijacking (9/23)...

Fri, 23 Sep 2011 22:16:25 -0400

Please review this link -

http://www.eff.org/deeplinks/2011/07/widespread-search-hijacking-in-the-us

Many ISP’s are redirecting your google.com DNS result to the IP Address of their “Paxfire” proxy server – effectively letting them record EVERY search term you Google for marketing or other invasive reasons.

This happens because most home users just use their ISP’s DNS servers and don’t think twice. I noticed this yesterday when Google refused to do any searches for me and a Google “captcha” came up (where you have to type in all the crazy letters to proceed) and gave a warning that my IP was sending many searches to Google (an IP, that traced back to DataPipe in San Jose, CA) and they had to make sure I was not a “virus or a bot” to use google.com.

Well I did some research… Instead of giving you the local search results or Google banner ads – ISP’s are trying to get their PAID SPONSORS on to the results page. This could also be used to harvest information for marketing reasons “John in Hoboken likes Kung fu Films and trips to Thailand”, etc.

The way to defeat this privacy invasion – make sure your home router has 3 neutral DNS servers that will not redirect your google.com session to the third party proxy (and these 3 are served via dhcp to your home subnet)

8.8.8.8 (google’s open dns)

8.8.4.4 (google’s open dns 2)

205.171.3.65 (Qwest business DNS resolver, doesn’t seem to proxy the google.com A record)

Thanks!

HSRP active router tells standby to take over when being rebooted

Wed, 21 Sep 2011 00:17:35 -0400

One cool fact I thought I would share – Understanding the HSRP resign message…

This evening I’m testing various failover scenarios and I have my primary (HSRP active) router (.2) the HSRP primary (for .1) on an external segment (where 2 ASA’s default to the HSRP virtual address (.11 , .10)

I don’t use standby tracking on the primary router – it’s simply tracking its local ISP connection default route via an SLA ping of the ISP adjacent neighbor on metro-E and track object. I have a backup default route, AD 254, to the Standby router (.3) on the segment. If the primary router is up, but has no valid default route (cause it can’t ping the adjacent ISP) it routes all traffic to standby router.

So I started my failover testing by rebooting primary router and doing a continuous ping of 4.2.2.2. Well, no pings were missed during the reboot!!! (I expected about 10 missed pings as the standby router became hsrp primary and took over .1, the default gateway of the ASA’s). (if you’re sharp you caught the fact we’re obviously natting the first public range to a second public range on the standby router to go out a different ISP)

Here is the debug output on the HSRP standby router during the primary’s reboot (watch for the “resign rcvd” where the primary RESIGNS and allows the standby to become active MUCH FASTER than the timers)

kcf1515-edge-rtr2#debug standby

HSRP debugging is on

kcf1515-edge-rtr2#

Sep 21 04:08:15.504: HSRP: Gi0/0 Grp 1 Hello in 174.58.91.2 Active pri 150 vIP 174.58.91.1

Sep 21 04:08:15.824: HSRP: Gi0/0 Grp 1 Hello out 174.58.91.3 Standby pri 100 vIP 174.58.91.1

Sep 21 04:08:18.824: HSRP: Gi0/0 Grp 1 Hello out 174.58.91.3 Standby pri 100 vIP 174.58.91.1

Sep 21 04:08:19.352: HSRP: Gi0/0 Grp 1 Hello in 174.58.91.2 Active pri 150 vIP 174.58.91.1

Sep 21 04:08:21.516: HSRP: Gi0/0 Grp 1 Resign in 174.58.91.2 Active pri 150 vIP 174.58.91.1

Sep 21 04:08:21.516: HSRP: Gi0/0 Grp 1 Standby: i/Resign rcvd (150/174.58.91.2)

Sep 21 04:08:21.516: HSRP: Gi0/0 Grp 1 Active router is local, was 174.58.91.2

Sep 21 04:08:21.516: HSRP: Gi0/0 Grp 1 Standby router is unknown, was local

Sep 21 04:08:21.516: HSRP: Gi0/0 Grp 1 Standby -> Active

Sep 21 04:08:21.516: %HSRP-5-STATECHANGE: GigabitEthernet0/0 Grp 1 state Standby -> Active

Sep 21 04:08:21.516: HSRP: Gi0/0 Grp 1 Redundancy “hsrp-Gi0/0-1” state Standby -> Active

Sep 21 04:08:21.516: HSRP: Gi0/0 Redundancy server “hsrp-Gi0/0-1” update, Standby -> Active

Sep 21 04:08:21.516: HSRP: Gi0/0 Redirect adv out, Active, active 1 passive 1

Sep 21 04:08:21.516: HSRP: Gi0/0 Grp 1 Hello out 174.58.91.3 Active pri 100 vIP 174.58.91.1

Sep 21 04:08:24.516: HSRP: Gi0/0 Grp 1 Hello out 174.58.91.3 Active pri 100 vIP 174.58.91.1

Sep 21 04:08:24.516: HSRP: Gi0/0 Grp 1 Redundancy group hsrp-Gi0/0-1 state Active -> Active

Sep 21 04:08:27.516: HSRP: Gi0/0 Grp 1 Hello out 174.58.91.3 Active pri 100 vIP 174.58.91.1

Sep 21 04:08:27.516: HSRP: Gi0/0 Grp 1 Redundancy group hsrp-Gi0/0-1 state Active -> Active

kcf1515-edge-rtr2#

Sep 21 04:08:30.076: HSRP: Gi0/0 Grp 1 Hello out 174.58.91.3 Active pri 100 vIP 174.58.91.1

Sep 21 04:08:30.516: HSRP: Gi0/0 Grp 1 Hello out 174.58.91.3 Active pri 100 vIP 174.58.91.1

kcf1515-edge-rtr2#un all

All possible debugging has been turned off

Joe

#19366

What planning goes into an "Actionable Disaster Recovery Plan"?

Mon, 15 Aug 2011 03:33:58 -0400

“Mission Protect” Background

Affirmed Systems designs and manages on-going disaster recovery environments for our Managed IT Services customers. The goal of this service is for clients to resume work in only a few minutes in the event their primary applications such as voice, email, financial applications, payroll and any business critical system are effected in an Emergency or outage of their primary technology facility.

Case Study

Financial Trading firm with collocated market access, internally developed trading software, market data feeds and other proprietary financial, tax and compliance software programs. There are currently 2 offices in the NYC Area and 1 office in Los Angeles, CA. The firm requires that in the event their headquarters NY Office is offline for more than 30 minutes the firm opens a nationwide conference bridge and follows a detailed IT and Logistics plan to resume normal operations. All staff are well trained and can work as normal from home or alternate business facilities.

The Solution

Affirmed Systems Mission Protect business continuity planning services.

Data replication and protection:

Throughout the business day, streaming real-time backup protection replicates database, file and other records to our warm standby disaster recovery private cloud infrastructure. Securely located in the Affirmed Systems collocation facility, all data and systems are managed and monitored by the Affirmed Systems Mission Protect specialist team. This protects critical data generated as a result of any business activity from outages at the primary site. In addition to replication of data, live standby systems are ready in minutes to facilitate continued access and operation. Only by providing both backup systems available from anywhere and your current data can Affirmed Systems insure total protection to our customers.

Voice systems and communication continuity:

In the event of an outage at the main facility, voice services including direct dial-in numbers, conference bridge service and other critical firm communication facilities are available on backup equipment at the warm standby facility. Within minutes of a communications effecting outage, staff are immediately reconnected to clients, customers and the world as normal. Communication facilities are again available from anywhere via multiple methods, including hardware devices, software and other mechanisms including call forwarding and message services. How your customers reach you all the time must continue as normal - even during an emergency to insure full operational capacity.

Staff training and company preparedness testing:

Affirmed Systems’ Mission Protect business continuity team works with your staff to insure proper training and a step-by-step plan to continue working in the event of a disaster. Who goes where? How do we access our systems and services that are critical to the business? Several times a year our team conducts tests with your staff to insure that everyone is ready, all systems are ready and your data is ready in multiple locations.

Get started today!

Affirmed Systems Mission Protect insures business continuity for critical applications, technology and communication systems.

Call Affirmed Systems today!

1 (866) 973-9933 x3 or email us contact@affirmedsystems.com

Affirmed Systems Managed IT Services Summer 2011 Special!

Thu, 14 Jul 2011 07:48:10 -0400

Affirmed Systems Managed IT Services Summer 2011 Special!:

Managed IT Services First month Free! Comprehensive Coverage includes: Monitoring, Maintenance, data backups and support.

Does your company have a Datacenter? Datacenter Managed Services: Outsource operations to experts! Call Today!

Thu, 14 Jul 2011 07:03:35 -0400

Does your company have a Datacenter? Datacenter Managed Services: Outsource operations to experts! Call Today!:

“Our customers bask in uptime, reliability and insight their competitors can only dream of” Joseph Brunner, CEO, CCIE 19366

IT Decisions faced by our customers...

Tue, 07 Jun 2011 10:32:00 -0400

As our customers build their business from an idea, to a small group to a larger corporation they are faced with many technology decisions. Some they face the day they incorporate. Some they may not face for many years and will rely on their “TRUSTED ADVISER” to make for them when they are ready…

So here are the top technology decisions our clients need to make and the pro’s and con’s of each…

1. Email hosting

Decision: Host email on in-house server vs. outsourced email hosting

Host email in-house

Pros: Fastest access to mailbox, especially for users with lots of email; secure as all email sits locally in the office and no outsiders could review or index it for targeted advertisements (Google does this with Google apps).

Cons: Need to purchase and maintain local in-house server; need to purchase licensing, maintenance and other fees. Backup, archive, spam, virus and storage issues. Also new user create/delete is an IT Process rather than a simple ticket or web portal. Possible confidentiality and performance issues, uncertain MTTR (mean time to repair) if down.

Outsourced email hosting

Pros: Anywhere access to email on fast, managed infrastructure. No local servers, no licensing costs, possible reduced complexity and operation (if provider is good)

Cons: Large mail stores are “in the cloud” not on Local Lan servers, requires outlook or mail suite in “cached mode” so many emails can be searched or reviewed when offline; SECURITY and other advertising issues, provider can harvest data from email and or charge for mail bandwidth. Possible lack of ability to restore sensitive or critical deleted emails – need clear contract terms on this for finance or similar firm.

2. Phone system

Decision: Use in-house phone system vs. outsourced IP phones

Utilize in-house phone system

Pros: Highest quality calling, ability to use local PRI T-1 voice lines, avoid complexity and static issues caused vy voip on Intenret; reduced complexity – no complex quality of service or dedicated internet lines for voice. Gain security of dedicated voice system.

Cons: Increased cost for MAC (moves, adds, changes); custom programming needed at local PBX or on-site Cisco voip switch each time a user changes, where a provider does this as a support ticket. Required in-house expertise required or local phone maintenance contract.

Outsourced IP Phones

Pros: Reduced cost, no need for local equipment, phone lines and other services on-site. Reduced long-distance calling costs, including international. Ability to provide home teleworkers IP Phone with ease.

Cons: Poor performance over congested internet links at unpredictable times – such during customer facing conference calls… Problem ownership issues (finger pointing) when voip doesn’t work. Voip Company blames network and internet provider, Internet provider blames Voip company, etc.

3. Server operating system virtualization

Decision: Standard Servers vs. Virtualized environment

Standard servers

Pros: Fastest performance to LOCAL Storage vs. overhead of virtualization; reduced licensing costs (only have to purchase local server operating system license). Reduced network and server networking complexity, less high performance network requirements of Jumbo Frames, iSCSI capable switches, etc.

Cons: Operating system can be corrupted easily, backup of server operating system images may require third party tool (Windows Server 2008 server backup does backup images of servers though). If Operating system down or needs to be restarted may require additional access device (such as Drac/ILo2) to reach system.

Virtualized environment

Pros: Ability to thin provision operating systems using “only” the disk space in use by files at actual time of install; Ability to share ram, disk resources (memory pooling, thin provisioning) between operating systems guests on virtual hypervisor… Ability to quickly “clone” operating system guests, providing rapid deployment of many similar servers (i.e. create 10 windows 2008 servers in minutes)

Cons: Requires complex, high performance networking and storage planning – optimal performance requires Local Hard Disk firmware optimization and selection process, network storage requires iSCSI TCP/IP offload engine (TOE) nic cards or Fiber Channel storage area network adapters (FCOE, etc). Possible speed and access latency caused by extra virtualization layer; in ability to use native device drivers on hardware in lieu of virtualized drivers – this can create issues in environments where programming is done to avoid hardware latency (i.e. kernel bypass routines)

4. Storage Area network Design

Decision: Choosing iSCSI vs. Fiber Channel

iSCSI

Pros: Able to effectively be supported on many low cost switches (i.e. gigabit Cisco C3750/E, HP Procurve Series with jumbo frames, MTU 9000). Able to use many iSCSI ToE (Tcp/ip offload engine) capable Broadcom and Intel nics that often come stock in most servers sold today. Easy IP routing allows for local and remote access to iSCSI storage using main ip network, inter-site replication over existing network hardware.

Cons: Most deployments are 1Gbps only (1/8 bits/bytes = 125MB, or about the speed of slower older drivers only when used over 1Gbps network). 10Gbps deployments still costly. Drive LUN lockup issues caused by network pathing to storage command path issues and blocking timeouts – result is possible freeze up and drive locking in guest operating systems. Requires iSCSI/vmware/Xen certified equipment to operate properly at production level speeds. Storage access is effected network protocols (such as STP, Routing convergence, ARP failures, etc)

Fiber Channel

Pros: Dedicated storage switching environment, with LUN Zoning, VSan fabric isolation and non-IP framing offered for high speed “uninterruptible” storage area network. Clear segmentation of networks used for Client Server facing, replication and backup network traffic vs. Storage Area Network. Network based options are emerging technologies such as FCoE (fiber channel over Ethernet) which maps fiber channel frames directly to Ethernet without layer 4/TCP-IP overhead of iSCSI.

Cons: High cost of dedicated SAN switches, high cost San controller interfaces and SAN software licensing is usually higher with Fiber Channel vs. iSCSI. Isolation of SAN environments on a site to site basis requires costly & complex network technologies such as FCIP (Fiber Channel over IP) capable devices to “route” storage frames for replication and data mirroring between Locations (where iSCSI just does this on standard network gear). Usually 2 to 4 times the cost of iSCSI for the same storage size deployment…

Joe

#19366

Affirmed Systems 2011 sales specials announced

Thu, 27 Jan 2011 05:22:00 -0500

Affirmed Systems 2011 sales specials announced

News Release
Source: Affirmed Systems Communications Director
On Thursday, January 27, 2011, 9:00 am EDT

NEW YORK, January 27, 2011— Affirmed Systems, the leading Managed IT Services firm, has announced 2011 sales specials designed to further accelerate competition and cost savings in the IT services industry. Affirmed Systems is offering new Affirmed Assurance Managed IT Service customers the first month of service free! Affirmed Assurance is a complete IT support, maintenance and back-up plan designed to empower business customers with their own affordable highly skilled IT department comprised of expert level engineers and systems staff. Providing remote monitoring, on-site support and 24 x 7 around the clock coverage, it has never been more affordable and easy to sign up! “We have proven Affirmed Assurance is the best Managed IT Services plan in the industry and now we are giving new customers the opportunity to start with their first month free. We have streamlined the on-boarding process, allowing companies to rapidly gain the coverage, services and quality we are known for”, stated Joe Brunner, Affirmed Systems CEO.

The second sales special, offers the free installation of all harware & software purchased from Affirmed Systems. “We noticed many of new customers presenting us with quotes they had received from other IT Companies for hardware sales and installation projects - with the installation service charges often running 30% to 50% or more of the total cost of network hardware, servers, desktops and other IT Hardware, itself often marked up above list prices. We said to ourselves “installing new hardware with common operating systems and configurations used by most businesses is a pretty straightforward process- something our engineers do so often they have it down to a science. Why not sell hardware at reasonable prices and include the installation for free?” It’s an honest, fair incentive to the customer to become a managed IT services customer, building a long-term, trusted relationship with us for the monitoring, maintenance and data back-up and recovery services that are the keystone of our business” stated Mr. Brunner.

Affirmed Systems is proud to be an Ingram Micro partner, HP and Dell Premier business reseller, providing customers with financing and payment options that put them in control of their technology investment budgets, at the same time insuring they receive the most modern, high performance hardware and software available today. “Affirmed Systems is driven by an innovative, expert team that always seeks the best and most affordable options for our customers. Our growth and success is really our customer’s growth and success. We do this for them” stated Chinyere Uba, the firm’s COO and co-founder. The Hardware sale free installation program covers the installation and configuration of routers, switches, firewalls, servers and desktops for VMware ESXi, Windows 2008 R2 Server, Windows 7 or Linux operating systems, installation of Microsoft Business Applications including MS Exchange 2010, MS SQL 2010, network printing, authentication and other standard configurations necessary to use a server or desktop in a corporate network. Custom configuration of non-Microsoft software, or other company specific applications may result in additional charges (to be determined by Affirmed Systems at time of hardware install and stated in writing to customer)

About Affirmed Systems

Affirmed Systems is the leading Managed IT Services provider, and datacenter services firm. Our clients “Invest in Uptime”, leveraging both our excellent customer service and top technology expertise. Monitored 24x7 from the Managed Services Operations Center (MSOC), our Affirmed Assurance program allows businesses to receive Flexible, fixed price per month IT Services for VoIP, Workstations, Servers and Network Systems.

Affirmed Systems core services are Datacenter Management, Managed IT Services, Technology project outsourcing, Network & Systems Security.

Guys that "know cisco" but say "the certs just prove you can take tests" (MYTH)

Thu, 23 Dec 2010 00:22:39 -0500

Ever wonder why getting vendor Certifications in Cisco, Microsoft, VMWare etc. are so important to your career as IT engineer?

Certifications with no Experience is of course bad! You need real experience working in the field to have at least a good understanding of how something works.

But often, experience with no certifications can be just as bad. Just because someone is “experienced” does not prove they have gained the knowledge necessary to really understand how something works. To really “prove” you got what it takes - get certified and get experience!

Getting certified helps you learn the theory behind how something works, so you can later understand exactly WHY YOU ARE doing something when working in the field.

Here is an example that comes to mind (I see this a lot):

I often see firewall/security guys list redundant statements in their firewall access lists. It’s obvious from reading these few lines of code some was either careless or simply does not understand what they are doing…

A few lines of a live access-list on a firewall;

access-list outside_allowed_in permit ip host 209.214.205.10 host 38.104.11.13

access-list outside_allowed_in permit gre host 209.214.205.10 host 38.104.11.13

access-list outside_allowed_in permit icmp host 209.214.205.10 host 38.104.11.13

What is wrong with this access-list?

Well the first line makes the second and third lines unnecessary!

I’ll explain…

“permit IP” states “all IP protocols, 1-255 are permitted” between the two hosts…

(source = 209.214.205.10, destination = 38.104.11.13)

The second line then states IP protocol 47, Generic routing encapsulation (GRE) packets should also be permitted. The third line states IP protocol 1, Internetwork control message protocol (ICMP) should be permitted.

GRE (47), ICMP (1), TCP (6), UDP (17), ESP (50), OSPF (89), EIGRP (88) are all types of IP packets (“network” or layer 3 of the OSI model).

Studying for and passing Certifications helps us at troubleshooting also! You’ll understand a lot more about what your are working on whether you have 1 day of or 20 years of “experience”.

(I removed the second and third lines in that ACL, and nothing broke :)

-Joe

#19366

Configuring SNMP for monitoring on ESXi 4.1

Thu, 16 Dec 2010 18:32:00 -0500

If you’re like most people you use the free edition of ESXi 4.1 server. Why not?

Most small businesses and companies do not need live motion, storage v-motion, and DRS and can live with a few minutes of downtime if an ESX server crashes (how often does this happen? LOL). The VMware licenses often cost more than the physical servers and therefore are out of reach of all but the plushest hedge funds, banks and financial exchanges, many of whom still balk at paying $10,000 to $20,000 to license ESXi features that buy them a few minutes more of uptime, about once every 2-5 years if a server dies :)

(We have ways to script re-start of vm’s on free edition of ESXi)

But as a Managed Services Provider we still must monitor every aspect of ESXi, including performance, availability, and the state of the virtual machines being hosted!

In order to do this, we must first enable the SNMP agent already available on every install of ESX 4 - 4.1.

Let’s get started-

1. Enable SSH remote management of the ESXi server from within the Vsphere client (Configuration, Software Section, Security Profile, Services, Properties -> Press “remote tech support (SSH)”) As shown

and choose the “Options” button. Make sure this is set to “Start Automatically”

Now, we can SSH remotely into our ESXi Host server and edit the following file

Using VI let’s edit the snmp.xml file with the following syntax (note: the target=the server’s IP where SNMP queries and walks will arrive, NOT the SOURCE IP where they will ORIGINATE)

Here’s a sample of the contents of snmp.xml so you can copy, edit and paste;

<config>
<snmpSettings>
<enable>true</enable>
<communities>public</communities>
<targets>10.21.21.12@161/public</targets>
<port>161</port>
</snmpSettings>
</config>

Now, let’s save the changes to our snmp.xml in VI

(use the :wq! write/quit option)

Last, let’s restart the vmware services using the “services.sh” script that is in the

/etc/vmware directory on ESXi

Now, using your SNMP string you can monitor your ESXi server from your NMS;

some common mib’s include

Monitor Name : Memory Used By VM
System OID : (.1.3.6.1.4.1.6876.3.2.4.1.4*100)/.1.3.6.1.4.1.6876.3.2.4.1.3

Monitor Name : CPU Utilization of VM
System OID : (.1.3.6.1.4.1.6876.3.1.2.1.3*100)/$DELTA_TIME

Monitor Name : CPU Utilization(UCD SNMP MIB)
System OID : .1.3.6.1.4.1.2021.11.9.0

Monitor Name : Disk Utilization
System OID : .1.3.6.1.2.1.25.2.3.1.6

Enjoy!

Understanding Cisco ASA firewall VPN tunnels (keep it simple when your just getting started part 2)

Wed, 27 Oct 2010 18:41:48 -0400

To continue our simplified vpn tunnel series, this time we are looking at Cisco ASA firewall vpn tunnels.

Here’s our basic config

ASA’s use nat 0 statements to bypass nat for traffic using the vpn tunnel.

! dont forget to deny nat for the private to private in your nat acl

access-list no-nat extended permit ip 192.168.1.0 255.255.255.0 192.168.2.0 255.255.255

(where 192.168.1.0/24 is the LOCAL network)

(! create crypto acl to match tunnel traffic

access-list to-site2 extended permit ip 192.168.1.0 255.255.255.0 192.168.2.0 255.255.255.0

! create phase 1 policy

crypto isakmp policy 10
authentication pre-share
encryption aes
hash sha
group 2
lifetime 86400

! configure pre-shared key (note your external vpn peer ip is specified)

tunnel-group 12.140.190.10 type ipsec-l2l

tunnel-group 12.140.190.10 ipsec-attributes
pre-shared-key s3Cr3T!!k3y

! configure phase 2 transform

crypto ipsec transform-set ESP-AES-128-SHA esp-aes esp-sha-hmac

! enable ike on the ASA’s outside interface (if not done already);

crypto isakmp enable outside

! configure crypto map, we’ll use sequence 100 so we can put some peers

! before this sequence should we have to later :)

crypto map outside_crypto_map 100 match address to-site2
crypto map outside_crypto_map 100 set peer 12.140.190.10
crypto map outside_crypto_map 100 set transform-set ESP-AES-128-SHA

! Apply the crypto map on the outside interface (our your outgoing interface name if you do not use the name “outside”)

crypto map outside_crypto_map interface outside

Major gotcha’s

1. Make sure your firewall uses the outgoing interface (the one that has the crypto map applied) to get to your vpn’s peer’s destination network (usually the default route satisfies this condition)

2. Make sure you do not nat vpn traffic (nat 0 statments must be defined in most cases)

3. Make sure no other existing vpn overlaps your destination network (i.e. in this example 192.168.2.0/24 as a destination is UNIQUE to site2.

-Joe

#19366

Issues with multiple users connecting to cisco IPSEC vpn from same location?

Wed, 27 Oct 2010 03:16:20 -0400

The scenario:

Recently, several of our clients have had issues when multiple staff members are working off-site at third-party locations. While on-site the users all connect back to our their vpn gateway via the Cisco IPSEC vpn client.

-Only 1 user at a time can connect to the Cisco IPSEC vpn client service.

-If a second user connects the first user’s vpn client is disconnected….

The Technical explanation of the problem:

By default, the Cisco IPSEC VPN Client uses ipsec over nat with UDP port 10000 as its transport. Many firewalls effect udp traffic (checkpoint, netscreen to name a few). Technically multiple users should be able to connect at the same time. Each user while inside the third-party network has a unique Private IP address and the firewall should be able to use UDP checksums and its flow table to keep track of sessions… However, often it appears the third-party’s firewall was getting confused, as the on-site users are all connecting via UDP 10000 to the same gateway IP on the internet (i.e. vpnny1.company.com)… Perhaps the third-party firewalls are unable to handle multiple simultaneous internal IP’s natting to the same external destination IP while using UDP, etc.?

As neither us or our client can do much to change the firewall configuration at another company’s offices or data centers (especially at Price Waterhouse LLP or Bank of America), here is a solution to the problem we can implement to quickly resolve the issue.

The Solution:

1. Configure the vpn gateway to accept ipsec over nat TCP connections (instead of UDP)!

(Example: ASA firewall)

crypto isakmp ipsec-over-tcp port 10000

2. Configure the user vpn client profiles to connect over TCP transport

-Open VPN Client

-Right click the vpn connection entry

-On Transport Tab, change UDP to TCP (radio button)

-Set port number to the port number you configured on your vpn gateway.

This fixes the issue in most circumstances. (Of course the port you choose will have to be allowed for use in the third-party company’s outbound firewall policies).

(Note: You can try to use TCP 443 if your supported users are at a very restrictive location, however you will lose the ability to manage the firewall on the default ASDM https port, TCP 443 due to conflict, however for management purposes the ASDM https port number can be changed as follows)

http server enable 4444

-Joe

#19366

Know when your vpn tunnels are down - configure Cisco IOS EEM email alerts

Wed, 01 Sep 2010 22:50:05 -0400

1. First let’s define some global variables the router will need to lookup the smtp relay’s address when sending email alerts (in our case we use Postini)

ip domain-lookup
ip name-server 4.2.2.2

2. Let’s get the sendmail.tcl script onto the local router’s flash. EEM applets will use this TCL script to send the actual alerts (script courtesy of and available on cisco.com)

smtdata-rtr-1#copy ftp://username:password@some.ftphost.com/sendmail.tcl flash:
Destination filename [sendmail.tcl]?
Accessing ftp://*****:*****@some.ftphost.com/sendmail.tcl…
Loading sendmail.tcl !
[OK - 5293/4096 bytes]

3. Let’s create an IP SLA and define the SLA as a tracked object in IOS. The SLA does the actual pinging and response checking, the tracked object simply monitors
whether the SLA is up or down.

Our goal here is to ping the inside IP address of our vpn peer router, sourced from our inside IP address. These source and destination’s will of course be part of the
crypto-acl and can use the tunnel; the SLA

ip sla monitor 10
type echo protocol ipIcmpEcho 10.1.1.1 source-interface GigabitEthernet0/1
timeout 2000
frequency 10
ip sla monitor schedule 10 life forever start-time now

track 10 rtr 10 reachability
delay down 10 up 10

4. Now let’s configure IOS Embedded Event Monitors to monitor the tracked object’s status in IOS, and as a result of the VPN tunnel going up or down alert us via email

! let’s define the EEM global parameters

event manager directory user policy “flash:/”
event manager policy sendmail.tcl

! let’s create our EEM applet that tracks the state of tracked object (which in turn tracks the reachability of the original SLA ping target) and when the vpn goes UP or DOWN

event manager applet colo-vpntunnel-up
event track 10 state up
action 1.0 info type routername
action 1.1 cli command “show track 10”
action 2.0 mail server “outbounds9.obsmtp.com” to “gekkoandco-notif@affirmedsystems.com” from “gekkoandco-rtr-hq@gekkoandco.com” subject “VPN Tunnel to Atlantic Metro From gekkoandco-rtr-hq Up @ $_info_routername” body “$_cli_result”

event manager applet colo-vpntunnel-down
event track 10 state down
action 1.0 info type routername
action 1.1 cli command “show track 10”
action 2.0 mail server “outbounds9.obsmtp.com” to “gekkoandco-notif@affirmedsystems.com” from “gekkoandco-rtr-hq@gekkoandco.com” subject “VPN Tunnel to Atlantic Metro From gekkoandco-rtr-hq Down @ $_info_routername” body “$_cli_result”

5. Last let’s not forget to put your router’s outbound public ip address into the permit list on your smtp relay