Senior UNIX System Administrator / Harris Interactive

In June of 2000, I made a decision to join the top internet-based market research firm of Harris Interactive as the Senior UNIX System Administrator for a venture billed as the "worlds largest interactive poll". It was called the "The Planet Project". In addition to Harris Interactive; 3Com, AT&T, SUN Microsystems, Oracle Corporation, BEA Systems Inc., Mercury Interactive, AAO productions, Alteon Web Systems, Akamai technologies, Macromedia, and others contributed facilities, hardware, software, or intellectual resources to the partnership. The production environment had two T3s to the Internet and spanned two production data centers in Silicon Valley interconnected by a pair of T1s. This environment was connected via VPN to the development data center in Rochester, NY. During the time I worked for Harris Interactive, my knowledge of Security, Infrastructure architecture, and large-scale multi-tier web sites grew immensely.
My role in the Planet Project was to research, design, implement, and provide ongoing support for the four-tier, highly available, web, application, operational database, and data mining server environment. For this time-critical implementation, I took the project infrastructure from empty racks to Alpha testing in approximately three weeks. I installed and configured: four E4500 data mining servers, two pairs of clustered E420R operational database servers, twenty SUN E420R application servers, twenty Netra t1405 web servers, a pair of secure SUN E450 log hosts and a pair of SUN Ultra 60 jumpstart / administrative servers. I provided guidance, and assisted in setup of: a NTP server, a Legato backup server, two pairs of SUN E450 DNS / SMTP and network intrusion detection servers, Alteon load balancing switches, Nokia firewalls, and 3Com layer three switches.
To facilitate speed of the setup, and consistency of the environment, I wrote Bourne shell scripts to automate and manage various parts of the pre, post, and server installation tasks using a custom Solaris jumpstart program. Prior to and during the event, the solution I created provided a fully functional first or second tier server in approximately 30 minutes. The custom jumpstart scripts: copied configuration files, installed packages, patches, and the operating system, mirrored disk drives, and configured primary, backup, and administrative network interfaces. I also wrote several administrative scripts that: updated custom applications, monitored CPU, memory, network, and disk I/O, and paged operations center if environments' operating thresholds were out of bounds.
I received guidance from some of the top security, performance, and network engineers/consultants from Deloitte & Touche, Metases, Eucid Inc., Euphorion and 3Com. The knowledge I gained about performance tuning and server security was applied in practice during the project. To prepare the environment for the large volume of expected hits, I tuned the kernel, TCP stack, network interfaces, and the applications. I secured the environment by: tightening vulnerable programs, only using Ssh encrypted connections to access the servers, and implementing remote host logging. For secure web access, I configured, compiled, and created a custom SSL version of Apache. During the load-testing phase of the project, I troubleshot CPU, RAM, I/O, and network bottlenecks.
After the Planet Project finished, I took over administration of Harris Interactive's UNIX infrastructure. This was comprised of three HPUX machines that generate ten million dollars of business quarterly and some sundry Linux and Solaris machines. I secured the environment by tightening down remote access, and fixing file permission problems. I migrated data between the servers to provide a more robust fail-safe environment, and I eased administration by implementing a multi-platform NIS domain. I implemented a popular monitoring and alerting system, and setup an on-line documentation library and knowledge base. I also managed Harris Enterprise HP Openview Network Node Manager implementation.



Infrastructure Analyst / Electronic Data Systems with SUN Microsystems

I worked on-site in SUN Microsystems Chelmsford, MA data center in an effort to transition support of it from SUN to EDS. I led the transition team in highly stressful and politically charged environment. Our main goals were:
  1. To support the servers during work hours, and provide on-call support after hours.
  2. To gather technical information about the servers and environment.
  3. To hire full time EDS employees to work on-site at SUN.
  4. To do this in an environment where many of the SUN staff were "less than pleased" about the turn of events.
For item 1: I supported SunPeak production server environment consisting of two SUN E10Ks, several SUN x500 class servers, and multiple infrastructure support servers. SunPeak is responsible for SUN's worldwide sales order fulfillment process, and during quarter end, is liable for approximately three million dollars worth of business per hour. I also created an on-call coverage schedule for EDS SA's for the duration of time we were there
To address item 2: I worked closely with SUN Microsystems 1st, 2nd, and 3rd shift system administrators to gain knowledge of the data center, and transition support. I researched and documented, the workflow processes, support methodologies, management and reporting tools, server environments, business organization contacts, and miscellaneous technical information that were unique to SUN Microsystems. I took this information and created an environment transition guide that was used to assist other engineers in working in the SUN environment. I presented this information on several occasions to EDS management and peers.
For item 3: I compiled a list of technical questions / answers, and served as technical interviewer for several prospective candidates that our management and recruiters were interested in.
To address item 4: I empathized with the individuals, as our team had similar experiences on the Xerox account. I focused on EDS' goals, was aware of the "political environment", did the best job I could, and presented myself in a professional manner.




Infrastructure Analyst / Electronic Data Systems with Xerox, General Motors