Hewlett-Packard

Hewlett-Packard - 1988-1990

Hewlett-Packard was my first full time employer post-graduation from RPI. I joined their Storage and Scanner division in Greeley Colorado, and at the time, it was mostly a lifestyle choice where I welcomed clean air, wide open spaces, and the opportunity to learn a long-lived corporate culture. John Young, the CEO, had also set a corporate goal of growing sales 10x in 10 years.

Every single employee was encouraged to create new product ideas, then work with someone on the marketing team, which was colocated in the same building, to estimate the annual recurring revenue, break even time, and return on investment in 1, 3, and 5 year scenarios. If the product looked good, the proposer would then take their metrics and use those metrics to recruit engineers who were working on other projects. There were no sacred cows. Even if a product was 6 months from shipping, engineers on that team could look at the financial metrics of another project and "jump ship" to work on the new product if they believed that the ARR, BET, and ROI were better for the company than the product they were currently on. So-called "managers" were officially facilitators instead of the traditional "command and control" common in most other corporations.

At the time I joined the Greeley division, their primary focus was tape storage and magneto-optic storage devices, which were write-once-read-many, and about 1GB per cartridge. All of the storage devices used SCSI and SCSI-II interfaces with the two main products being (1) a standalone drive which managed one cartridge, and (2) a robotic auto-changer that had 2 internal drives and 32 shelves for cartridge storage. All R&D engineering and marketing offices were on the second floor of the building, and the first floor was manufacturing and what were called "manufacturing engineers".

Among my first assignments at HP was quite a bit of classroom training on techniques and processes that HP found valuable to their engineering culture. I became certified as a 6-sigma coach, and took a deep dive on TPS (Toyota Production System) and their tools such as Ishikawa diagrams. I also became immersed in preventing defects at all stages in an R&D process, with the main summary of the cost of fixing defects being exponentially more expensive the earlier those defects were introduced into the development process. IE: a requirements defect cost 10x a high level design defect, a high level design defect cost 10x a low level design defect, a low-level design defect cost 10x a coding defect, and a coding defect cost 10x an automated test defect, with the end result that a requirements defect (missing or misunderstood requirement) would cost 10,000x a defect in the automated tests. One of the primary tools for eliminating defects in requirements was cross-functional team meetings to understand and review the requirements before even designing a solution. The cross-functional teams would include R&D engineers, marketing, manufacturing engineers, sales office staff, and field engineers.

My initial product development focused on device drivers for HPUX workstations (HP9000) and MPE (HP3000) mini-computers, which I chose because nobody else in the entire building had experience with direct access storage device drivers for preemptive multitasking operating systems (their experience was with tape drives, which were sequential access and did not allow concurrent access from multiple applications). I designed and implemented the DASD drives for HPUX in the "C" programming language, the DASD drivers for MPE in "Pascal", and also had to incorporate the DASD drivers into the HP9000 boot roms (using PASCAL) so that the workstations could boot off of the SCSI DASD devices. My drivers were a complete implementation of SCSI and SCSI-II initiator, as well as a SCSI-II initiator for autochanger control. The trickiest part was determining the autochanger state from a power on start without damaging the mechanical parts in the autochanger.

At some point, my direct manager learned that I was coming to office each day at 5am, and often leaving after 8pm. For a few weeks, he would stop by my office every day at 1pm and ask me what time I started -- and he looked at his watch, and sent me home on the spot. Once I asked him why, and his answer was that I had already put in an 8-hour day, and he shared an expression "are you living to work, or working to live?" I enjoyed the next few months having two full days of experiences every day -- 8 hours at work, and 8 hours of sunshine (in the summer) to enjoy outdoor activities and drive into the mountains to explore. He also forbade me to come in on weekends and enforced all of my "overtime" as comp-time to be taken off later.

Once the drivers were completed and handed off to the manufacturing team, I commenced working with another engineer who was designing an in-house VLSI for SCSI-II target devices in order to replace 3rd party chipsets which were not highly integrated and took many square inches of PCB space. Our storage division had not previously implemented a VLSI chip, and my previous experience with ASICS (for Myarc) and VLSI design teams at RPI were invaluable for the initiative. I rewrote a complete SCSI-II target device drivers for both DASD and autochanger, and identified portions of the code and overall architecture which would benefit from hardware implementation. We completed the net-list for the SCSI-II VLSI (we were not using a higher level tool such as VHDL or verilog) and I wrote the complete set of test vectors for verification of functionality. We sent the design to fabrication and it worked on the first tape-out without modification.

Near the end of my first year, I got involved in another project with writing custom HPUX drivers for the tape storage devices. We had received a request from NASA who had a warehouse full of tapes of telemetry and other flight data from the Mercury and Apollo missions. NASA simply wanted to read those tapes and store the data on more modern storage devices where the data could be analyzed. The first hurdle for each tape was identifying the device that had written it. During the 1960s and early 1970s there were dozens of non-standardized ways of writing tape, with the primary variable being the manufacturer of the device. Encodings included 7-bit, 9-bit, NRZ, Phase-encoding, manchester coding, diagonal domains, among others. The various computer systems of that day and age were not generally interoperable in the sense of taking a tape from one system and loading it on another. Our strategy was to use our latest generation of tape drive, override its internal data separator, and send high frequency samples of raw data to the HPUX workstation for decomposition, analysis, and ultimately conversion back to computer readable data. I developed a separate software analysis and decoder for each manufacturer we found in the tape library, and a plug in architecture wherein future engineers could write support for more manufacturers. There were other problems with the tapes from the warehouse -- they had been left, for decades, without the routine preventative maintenance required for tapes to have a long life -- they had not been retensioned, and many were damaged by water or humidity. The lack of retensioning caused a side effect known as "Write through", where the tape's magnetic programming would gradually transfer to adjacent layers of the tape on the reel, and effectively scramble the data. A lot of times, we were able to detect and remove write-through, because the amplitude of the magnetism from the write-through was much lower than that of the correct data, but we still needed to inspect and manage thresholding and quantization of the analog samples before converting to digital. Even then, there were cases where the parity and error correction codes reported data corruption. The water damage was worse, and many tapes could not be recovered with the read heads and a/d conversion we had at that time. The end of the project was a sad story -- NASA, even using our best tech at the time, was only able to recover data from twenty percent of the reels of tape in that warehouse.

In December 1999 me and 2 others on my team were tasked with integrating our storage devices (tape drives, magneto optic drives, and autochanger) with the workstation of Apollo Computer which had recently been acquired by HP. Rumor was that HP had acquired Apollo after losing the bid on a major workstation deal with Boeing, wherein Apollo had won Boeing's Business for mechanical CAD workstations. Apollo's offices were near Boston Mass, and our team was based in Greeley. Between the 3 of us, we devised a work schedule as follows: two of us were onsite and on duty every week, while the 3rd took a week off. The net effect was working long hours for two weeks, then having a week off (generally back in Colorado). We rented a townhouse near Apollo and started work. Since the entire Apollo operating system was written in Pascal, my previous Pascal drivers from MPE and the HP9000 boot roms was an invaluable start to the project. We integrated those drivers into the Apollo "Domain" OS, while at the same time learning their software engineering environment, the Domain Software Engineering Environment ("DSEE", affectionately "dizzy"). DSEE had integrated project management features and version control that I have not seen anywhere else in the past several decades. The DSEE version control was fully integrated into the filesystem and the file naming convention, unlike the GIT repos common today. I am informed and believe that the DSEE team left in disgust after the HP acquisition (they were not feeling valued) and founded "Atria" software which developed ClearCase, and ClearCase was ultimately acquired by IBM and is still in use to this day. ClearCase did not include the advanced project management features from DSEE. Our initial implementation of our drivers on DOMAIN OS was completed in the first month, with one exception -- every once in a while, the entire workstation would hang for 10-15 seconds when accessing the auto-changer. It was clear to me that the operating system was holding some sort of mutex across the autochanger seek operation, a contention hotly denied by the Apollo Domain team, who kept insisting that it had to be an issue in our code. I disagreed with them and set out to find the source of the issue. Inside of the autochanger "seek" functions I added some custom stack walking and "reflection" code which could detect mutex locks referenced by pointers on the stack, and quickly found the culprit completely outside of our new driver code. I patched and tested the source code of the DOMAIN OS to coordinate the mutex across seeks and submitted the code to the in-house Apollo OS team after I tested it. They were still in denial about their synchronization defect until I called a meeting and showed them the working version I had created.

I made the decision to leave HP and join Chase Manhattan bank in mid-1990 for a bitter-sweet set of reasons: 1) I had learned that promotion and advancement in salary at HP's Colorado divisions, during my time there, was not based on merit, but was based on seniority, and that one could only move up in salary or official responsibility if a vacancy was created above you in the org chart, and 2) I was invited by a friend of mine to join him at Chase Manhattan in New York where I had other personal interests.

SCSI-II
6-sigma
Requirements before Design
HPUX drivers 68020 "C"
Boot from SCSI - Pascal
MPE drivers "Pascal"
ASIC/VLSI, test vectors
SCSI-II Autochanger drivers
SCSI-II magneto optic drivers
Apollo Computer Acquisition - 2 weeks on, 1 week off
Pascal
Long locks held across autochanger seeks
NASA
1350 acre park
Apollo - Domain - SCSI-II Autochanger drivers
Apollo - Domain - SCSI-II magneto optic drivers
Apollo DSEE design software - Atria Software -> ClearCase -> IBM