the operator does not seek to encourage moves between underutilized clusters 5) φr(100% utilization) = kφr(0% utilization), for some constant k, to limit the impact on the initial endowment of budget dollars
The inputs of the weighting functions are utilization per- centiles for the different resource dimensions (e.g. CPU, RAM, disk, network). The fifth property is strongly related to the strategy used for disbursement of initial budget dollars among bidders and is not covered here. Figure 2 shows some example weighting functions that were used in our market experiments.
V. EXPERIMENTAL RESULTS
Building a market economy inside a commercial entity requires an extensive commercialization stack separate from the distributed applications and cluster management systems that actually use these resources . In particular, the ideas from the previous sections were used as the basis for decision support, price formulation, and trading support layers of an experimental resource allocation economy within Google. Other important systems for accounting, billing, and contract management are not discussed here as they are outside the scope of this paper.
To build the trading system, each resource pool was taken as a cluster / resource type combination with the latter including CPU, RAM, and disk. Engineering teams were given budget dollars and allowed to buy, sell, and trade resources with each other as well as the company itself. These transactions were priced by running periodic clock auctions with utilization- weighted reserve prices of the type described above.
In the remainder of this section, we discuss the various pieces of this system and then evaluate the pricing and provisioning outcomes produced.
A. The Trading Platform Implementation
The trading platform front end was implemented as an internal web application that could be used by teams to express trades in a simple bidding language against the available clusters participating in the auction. Users are greeted on a main page “market summary” (see Figure 3) that lists the participating clusters along with the number of active bids and offers in each, and the current market prices as determined by the clock auction.
Market participants enter bids through a two step process where they first enter requirements in terms of desired cluster resources (such as GFS  or Bigtable  resources). The second step, as shown in Figure 4, displays the covering amount of CPU, RAM, and disk and the current market prices for those components before allowing the user to enter a maximum bid price (for bids) or a minimum reserve price (for offers).
Given these bids, the trading platform then maps these into a simulated clock auction of the form discussed previously. Each engineering team generally corresponds to one “user” in the latter, although teams may be mapped into several, independent “users” if they are entering unlinked requests for
Template for “market summary” page on trading platform front end.
Template for bid entry page.
resources across multiple clusters. In addition, the company itself may be mapped into clock auction participants if it is willing to buy or sell resources in one or more of the pools.
The parameters of this environment are fed into a separate clock auction simulator. The latter, written in Python, runs the clock auction algorithm to determine winning prices and allocations for the (simulated) users. This mapping, simulation, and price update process is run at periodic intervals during the bid collection phase; the preliminary, updated settlement prices are displayed on the market front end, as shown in Figure 5 above. At the conclusion of this phase, one last simulation is run. The results of this are used to determine the final, binding market prices and engineering team allocations.
B. Trading Activity
At the time of this writing we have run six, experimental auctions over the course of several months. As desired, we have seen excess demand raise the price of resources which were previously oversubscribed and seen a number of groups move to less crowded clusters. The plot in Figure 6 shows the ultimate settlement prices for one of our first auctions as a ratio over the former fixed price that was in place before the market economy. The boxplot in Figure 7 shows the utilization percentile of settled trades in the auction broken down by