<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
Scott Dowdle wrote:
<blockquote
cite="mid:20696513.1871221924597424.JavaMail.root@mail.montanalinux.org"
type="cite">
<pre wrap="">Robert,
I've finished up the interview questions and you can find theme here:
<a class="moz-txt-link-freetext" href="http://www.montanalinux.org/files/robert-nelson-interview.html">http://www.montanalinux.org/files/robert-nelson-interview.html</a>
</pre>
</blockquote>
<p><b>Who is Robert Nelson?</b>
</p>
<p><i>Please answer whatever questions you feel comfortable answering.</i>
</p>
<p><b>Q:</b> Please tell me a little bit about yourself. Where are
you from (originally and now)? What is your educational background?
What are your hobbies? What is your family status (married, kids?)?
What do you do for a living?<br>
</p>
<p>I was born in Burnaby, British Columbia, Canada. Burnaby is a
suburb of Vancouver where Expo '86 was held. I spent most of my life
living in various cities all across Canada. In 1992 I moved to
Seattle, Washington in the United States to work at Microsoft. In 2004
I retired from Microsoft and currently live in Bellevue, Washington a
few miles from the Microsoft Campus. My partner of 8 years and I don't
have any children but we have two miniature Dachshunds that think they
are our children.<br>
</p>
<p>Since retiring I've occupied my time managing my real estate
investments and contributing to open-source projects, programming is
probably the closest thing I have to a hobby :-). Most of the
open-source projects I've been involved with have been directly or
indirectly related to my business.<br>
</p>
<p><b>Q:</b> How long have you been programming? What programming
languages do you use / prefer? Are there any other software projects
you are / have been involved with that you would like to mention?<br>
</p>
<p>I started programming in 1973 when I used to skip high school to
sneak off to Simon Fraser University to play with the IBM 370/155
mainframe. I was around so much that they offered me a summer job
developing courses using a CAI language they developed as an extension
to APL. So my first languages were APL/CAI and CourseWriter III. From
there I branched into PL/1 and System 360 Assembler.<br>
</p>
<p>Over the years I've learned and programmed in pretty much every
programming language ever developed, including some oldies but goodies
like Fortran and Cobol (Somehow I missed out on Algol). Most of my
professional life has been spent programming in C/C++ and various
machine languages.<br>
</p>
<p>Lately I've been working a lot in Perl, PHP, Python and Shell
scripts because they are the primary languages used in open-source
projects.<br>
</p>
<p>I don't really have any preferences regarding specific programming
languages, I believe that they are all just tools to get the job done.
Some are better suited for certain jobs than others but they all have
strengths and weaknesses.<br>
</p>
<p>Ever since I started playing with Actor (a defunct language, like
Smalltalk but with a syntax similar to C++ rather than Pascal), I found
I prefer object-oriented ones. I find that object-oriented programming
is the best way to organize my thoughts and make large projects more
manageable. Even when I'm using straight C code I still organize the
code as if I was writing C++.<br>
</p>
<p>For most of my career I've worked on "system code", operating
systems, compilers and device drivers. When I was with Motorola I
worked primarily on proprietary and Unix SVR4 minicomputers. At
Microsoft I worked on the Interactive TV project, Windows CE and in the
Windows NT kernel group on the I/O subsystem and Plug and Play. Since
leaving Microsoft I've worked mainly on Linux. I avoid a religious
attachment to any platform, I feel that, like programming languages,
each has its own strengths and weaknesses. Sometimes I think that the
acolytes on all sides must be compensating for some physical
shortcoming of their own. :-)<br>
</p>
<p>The main open-source projects I've contributed to include Bacula,
mtx, FreePBX, and GForge. I've also contributed fixes to countless
others. I've contributed a few of the tools I've written to the
open-source community. Usually my involvement starts out with fixes
for bugs that hinder my use of the software. If there is some area
that could be improved to make the software much more useful for me
then my contribution might be larger. It really depends on my interest
and how easy it is to work with the other developers involved in the
project. But my involvement is usually selfish.<br>
</p>
<p><b>Q:</b> How long have you been using OpenVZ? What other
virtualization products have you tried and do you use? What do you use
OpenVZ for?
<br>
</p>
<p>I've been using OpenVZ for over a year. My interest in
virtualization products sprang from my desire to get more use out of a
dedicated server I leased and my work on mtx (which I took over about a
year or so ago) which in turn was an offshoot of my involvement in
Bacula. Both Bacula and mtx required building and testing on a wide
variety of operating systems and versions. I found the process of
installing and booting all those operating systems tedious and was
looking for a better solution than filling my house with dedicated
machines.<br>
</p>
<p>I started out with VMware. While it mostly met my needs, I prefer
an open-source solution where I can change the things I don't like and
there is more likely hood of someone contributing other useful tools.
I then switched my focus to Xen. It provided most of the functionality
of VMware albeit with reduced performance for Windows guests due to a
lack of PV drivers for video and disk. <br>
</p>
<p>That would have probably been the end of my virtualization quest
were it not for another requirement coming from my business. Since a
number of my investments are in Canada I've been using VoIP for a few
years. I had been running an Asterisk server in my house but found
that Comcast's network somewhat unreliable in terms of latency. So
after shopping around I found a good deal on a dedicated server with
great connectivity and moved my Asterisk server there.<br>
</p>
<p>Based on the success of running the server there I decided it would
be nice to move other servers like my email server out of my house and
on to the server. The only drawback was that the system was somewhat
resource limited and the cost of increasing memory was a significant
monthly increase. So I went looking for a more resource efficient way
of running multiple virtual servers.<br>
</p>
<p>I looked at VServer and OpenVZ, VServer seemed unreliable and
without much of a community backing it up. OpenVZ fit the bill and I
settled on it. It has worked well on my dedicated server. I run three
virtual machines, a DNS server, an Asterisk server and a Zimbra
server. I've since replaced the rented dedicated server with a
co-located one of my own and added two additional virtual servers
Funambol for mobile sync and EJBCA for a certificate authority.<br>
</p>
<p>As a result of my experience with OpenVZ I set up a virtual build
machine that runs about 16 variations of operating systems and versions
using a combination of Xen and OpenVZ. This allows me to a release a
new version of mtx for for two versions of Debian, three of Fedora, two
of CentOS/RHEL, two of FreeBSD, three of OpenSUSE, three of Ubuntu and
Windows for both 32 and 64 bits in an hour or so. That's a total of 32
different builds all using one machine with no reinstalls, rebooting or
other manual steps.<br>
</p>
<p><b>The stock vzpkg</b>
</p>
<p><b>Q:</b> So, vzpkg used to work fairly well but over time, in
certain situations, it started to fail. What is wrong with the current
version?
<br>
</p>
<p>The main limitation of the current version is it was developed to
support RedHat distributions and is dependent on Yum/RPM. Another
limitation is that, due to the structure of the template meta data,
there was a lot of duplication of information resulting in extra
maintenance.<br>
</p>
<p><b>vzpkg2 and pkg-cacher</b>
</p>
<p><b>Q:</b> You have added a number of features / capabilities to
vzpkg. Could you give us an overview of what's new?<br>
</p>
<p>I think the most significant change over the stock version of vzpkg
is the separation of the packager specific code from the higher level
code. This allows scripts to be written to support other package
managers like apt which is used on Debian and Ubuntu. <br>
</p>
<p>The other slightly less significant change is the introduction of
the concept of a hierarchical structure to the template meta data.
Information which is the same for all versions and platforms of a
distribution need only be specified once. If there is a need for
separate settings for a specific version it can be overridden by a file
lower in the template meta data tree.<br>
</p>
<p>Also new packager-independent commands have been added for managing
packages in installed containers.<br>
</p>
<p><b>Q:</b> You added a new package named pkg-cacher. Where did
pkg-cacher come from and what does it do exactly? Can pkg-cacher be
used independantly of vzpkg2?<br>
</p>
<p>Most people managing multiple machines (physical or virtual) end up
installing a local mirror of some sort. This ranges from a subset of a
distribution like only the updates for a distribution on a single
platform to multiple mirrors of multiple distributions, versions and
platforms. These mirrors are generally maintained using rsync. They
are used to reduce bandwidth usage and installation time.<br>
</p>
<p>However the amount of disk space and bandwidth used to maintain
these mirrors can be quite significant. Particularly when most of the
packages are never actually used in the target environment.<br>
</p>
<p>While looking for a solution to these drawbacks I came across
apt-cacher available with Debian. It is a server written in Perl that
acts as a transparent caching HTTP proxy. It processes requests from
apt just like an HTTP server but forwards them to a distribution server
and keeps the results in a cache, then uses it to respond to subsequent
requests for the same file. It knows about the different types of
files: packages versus packager meta data. Since a package is static
once released there is no need to check the server for updated versions
whereas meta data changes over time and the distribution server must be
checked for updates. It also understands that the data may be present
on multiple mirrors but the content will be the same regardless of
which mirror is used.<br>
</p>
<p>General purpose caching proxies such as Squid may be used but they
do not understand the unique attributes of distribution repositories
and will duplicate files retrieved from different mirrors. They also
rely on the HTTP headers to decide retention policy rather than using
the packager meta data.<br>
</p>
<p>I used apt-cacher to handle my Debian distributions but wanted the
same functionality for my other distributions such as RedHat derived
ones. So I rewrote it as pkg-cacher.<br>
</p>
<p>Apt-cacher takes advantage of a key property of the Debian
distributions. The version and platform specific meta data is stored
separately from the packages. The packages for all versions are stored
in a single consolidated set of directories so there is no chance of
two packages having the same file name but different content.<br>
</p>
<p>However the same is not true of RedHat derived distributions. Each
version and platform has its own copy of the packages built using that
release and there are a number of packages with identical names but
different content. There are also packages which are unchanged from
version to version within a distribution as well as across
distributions.<br>
</p>
<p>In order to deal with these differences pkg-cacher uses a different
directory structure for its cache.<br>
</p>
<p>Other significant differences from Debian are the RedHat packager
uses the Range HTTP header to retrieve partial information from the
packages and some distributions use the HTTP Redirect header to
transfer to a mirror closest to the client. I have added support for
these headers in pkg-cacher.<br>
</p>
<p>Pkg-cacher is designed to be a standalone tool separate from the new
vzpkg2. However its use complements vzpkg2 and the default
installation of vzpkg2 depends on it.<br>
</p>
<p><b>Q:</b> How does pkg-cacher enhance vzpkg2?<br>
</p>
<p>The original vzpkg reduces the downloads by pointing yum's cache at
a directory within the template meta data tree. While this was a step
in the right direction, it still meant duplication across platforms.
It also provided no benefit to installed containers.<br>
</p>
<p>Pkg-cacher provides the benefits described in the previous question
for producing cached templates as well as installed containers.<br>
</p>
<p><b>Q:</b> Does pkg-cacher come into play from the perspective of the
containers?<br>
</p>
<p>The default templates included as part of vzpkg2 configure the
template meta data so that it uses the pkg-cacher server configured in
vzpkg.conf as VZPKG_CACHE_HOST. The operating system installed
configuration files are disabled by renaming them with a .disabled
suffix and a new configuration file is installed pointing to the
pkg-cacher server.<br>
</p>
<p><b>Q:</b> What container configuration changes have to be made in
order for a container to use the services provided by pkg-cacher?<br>
</p>
<p>Generally all that needs to be done is change the name of server in
the packager configuration files. This done automatically for
containers installed from cached templates generated by vzpkg2.<br>
</p>
<p><b>Q:</b> Does one have to use pkg-cacher in order to use vzpkg2?<br>
</p>
<p>No, all that is required is modifying the vzpkg.conf files located
in the template meta data to use another proxy server, the original
distribution servers or even a copy of the distribution server in the
local filesystem.<br>
</p>
<p><b>Q:</b> Are there any features in vzpkg2 that you can't use
without pkg-cacher? <br>
</p>
<p>No, pkg-cacher supplements vzpkg2 providing more efficient use of
resources.<br>
</p>
<p><b>A vision of the future</b>
</p>
<p><b>Q:</b> What is the next step? After vzpkg2 has had a bit more
community testing and you have gotten feedback and made any additional
changes to it that are needed, is the plan for it to replace the
official vzpkg or would you prefer it to stay an independant / separate
app?<br>
</p>
<p>I would like to see the vzpkg2 changes incorporated into OpenVZ and
replace the current outdated vzpkg. I anticipate that pkg-cacher will
always remain a separate tool because of its general usefulness.<br>
</p>
<p><b>Q:</b> Are there any features you haven't added to vzpkg2
(or pkg-cacher) yet that you hope to impliment in the not too distant
future?<br>
</p>
<p>There are still a number of features of apt-cacher which I haven't
rewritten to work with pkg-cacher. These are primarily in the area of
maintenance of the cache, such as removal of packages which are no
longer referenced by the packager meta data. I also plan on
eliminating the dependency on the lockfile utility included as part of
procmail. This was a dependency that I didn't realize was there until
it was brought to my attention recently. The final planned change is
conversion from a multiple process to a multithreaded application for
improved efficiency. A minor configuration change is the port that
pkg-cacher uses. It currently uses port 3142 which apt-cacher used.
However that port is actually registered to something else but near as
I can tell isn't actually used for its intended purpose. In the
interest of being a good netizen, I currently have a registration
request pending with the IANA for a port specifically for pkg-cacher's
use..<br>
</p>
<p>For vzpkg2 there are two significant remaining work items. First is
the completion of the manual pages for all the commands. The second is
support for a way of specifying the included packages incrementally.<br>
</p>
<p>Currently package lists further down in the template meta data tree
replace those above. Ideally it would be useful to say for example
that, in a specific version, package X shouldn't be included but
package Y should. Also it would be nice to be able to include the
processed list of packages from another list. For example you would be
able to say that the "web server" configuration is all the packages
included in the "small" configuration with packages X, Y and Z added.<br>
</p>
<p><b>Q:</b> What limitations, if any, do you see with vzpkg2 and what
would your perfect OS Template manager be like?<br>
</p>
<p>I think the only significant limitation of vzpkg2 is its
implementation using shell scripts. Some functions would be much
easier to implement in Perl or Python. As far as functionality I
believe that, with the additions described in the previous answer, it
addresses all my needs for a package manager. I'm happy to entertain
any suggestions others might have. <br>
</p>
<p>I suppose the obvious area that others might take exception to is
the lack of support for either OpenSUSE or GenToo. For OpenSUSE I've
created templates for version 10.x however in 11.x the OpenSUSE folks
modified rpm in way completely incompatible with the upstream version
as well as the version supplied on every other distribution. They did
this for an increase in compression ratios whose benefit is far
outweighed by problems caused by the incompatibility with the rest of
the world and even previous versions of their own distribution. They
created this incompatible version of rpm without renaming the tool.
Because of this incredible lack of judgment on their part (IMHO) I
haven't bothered to support it. That doesn't stop anyone else, with a
need, from porting their version of rpm to other distributions,
renaming it something like rpmSUSE and creating the appropriate
packager specific scripts for vzpkg2. That is one of the main benefits
of vzpkg2 over vzpkg is it is extensible merely by adding additional
scripts specific to the packager used on the new distribution.<br>
</p>
<p>There is no technical reason why GenToo couldn't be supported by the
current vzpkg2 other than I just haven't gotten around to writing the
interface scripts for emerge. I concentrated on yum/rpm and apt/dpkg
to ensure that I had the right level of abstraction to deal with two
very different packaging solutions. I also figured that supporting
RedHat and Debian based distributions covered the vast majority of
users.<br>
</p>
<p><b>Q:</b> Given the vast amount of changes from vzpkg to vzpkg2,
and the addition of pkg-cacher... I see a need for a bit of updated /
additional documentation. How is that going? All done? Need some help? <br>
</p>
<p>Documentation is one area where there is always need for more,
better, more concise, ... (fill in your favorite adjective here). I am
working to create additional manual pages for all the commands. But if
someone wants to volunteer, particularly on creating higher level, user
friendlier docs I think that would be great.<br>
<br>
</p>
<p><b>In conclusion</b>
</p>
<p>Robert, thank you... for your work on vzpkg2 and pkg-cacher...
and for the time you put into answering my quetsions. One last one for
you though...
</p>
<p><b>Q:</b> Are there any topics I've overlooked that you'd like to
mention or any additional comments you'd like to make?
</p>
None that I can think of.<br>
<br>
</body>
</html>