Merge branch 'master' of

Nico Schottelius 3 years ago
commit a7f1f52713

@ -0,0 +1,69 @@
title: How i run my ceph cluster or how you can make a killer storage solutions for almost free
pub_date: 2020-01-08
author: ungleich
twitter_handle: ungleich
_hidden: yes
_discoverable: no
I wanted to store some data data and this is what I've came up with.
Low cost, high tech datastorage with ceph
First of all why would you run a ceph cluster? Its complex, a bit time consuming and easier to lose data than using zfs or ext4.
My reasons:
its very easy to expand/shrink
manage all your data/disks from 1 host (can be a security risk too)
its fun
we have it in production and it scales well
unifying the physical
Step 1 :
Find your local hw dealer.
Second hand sites can be a good source, but good deals are rare. My tactics on is: server & zubehor, filters: used, auction, max 1,5kchf, sorted with the ending soonest first) link :
Nearby dangerous material (ewaste) handler companies can be a goldmine. Big companies cannot just throw used hardware out as regular waste becuse electronics contains a little amount of lead (or some other heavy metal). So big compnies sometimes happy to sell it as a used product for cheap and ewaste companies are happy if they get more money than the recycled price / kg which is very-very low
low quality (core2duo era) pc-s also suffice, but you wont be able to do erasure coded pools are they use a ton of processing power and ram.. be careful with the ram, if you run out of swap/ram, your osd process will be killed, learnt it the hard way. also sometimes recovery uses more ram than usual so keep some free ram for safety.
Put 10G nics on your shopping list, for performance, its absolutely crucial. I've started without it, and its certainly doable but it wont perform well. A little hack is to pick up gigabit nic cards (as some people give them away for free), and put them in an lacp bond. Note here: lacp doesnt make a single connection's speed better, the benefit is only realized at parallel connections.
If you dont care or have equal disks by size or speed no worries, ceph will happily consume everything you feed to it (except smr disks* or strictly only for frozen data) One hack is to snag some old/low capacity disks for free. If you do everyting right you can surpass ssd speeds with crappy spinning rusts. Worried about disks dying? Just have higher redundancy levels (keep 2 extra copies of your data)
My personal approach is to have coldish data 2 times, hot data like vms 3x and 1 extra copy of both on non-ceph filesystem.
You can also group disks by performance/size. Ideally the disks should be uniform in a ceph device class, and equally distributed between hosts.
Avoid hardware raid, use cards that allow full control for to os over the disks. If you must use hw raid, raid0 is the way.
You can check out my ugly install script that meant to bootstrap a cluster on a vm.
tested on an alpine vm with an attached /dev/sdb datablock (don't use legacy ip (ipv4))
apk add bash
bash ./ceph-setup $ip_address_of_the_machine $subnet_that_you_will_plan_to_use
I've never prepared a disks manually yet which i should definetly review, because Nico wrote amazing helper scripts which can be found in our repo:
Some scripts still need minor modifications because alpine doesnt ship ceph init scripts yet. For the time being I manage the processes by hand.
Alpine's vanilla kernel doesnt have rbd support compiled in it atm, but for any kernel that doesnt have the kernel module, just use rbd-nbd to map block devices from you cluster.
Some useful commands:
Today were at Leipzig, Germany. Last day of [36c3](, Chaos Communication Congress.
## The earth is getting hotter

@ -0,0 +1,73 @@
title: Visit a zero carbon data center
pub_date: 2020-01-07
author: Nico Schottelius
twitter_handle: NicoSchottelius
_hidden: no
_discoverable: yes
Don't trust us - trust your eyes
From time to time we get the question "Are you really a zero carbon
data center? How do you proof it?".
As we use a local hydro power plant and local solar panels, we don't
have CO2 compensation certificates, because, well, we don't
compensate, but don't emit in the first place.
## Don't take our word
But why should you believe us? Maybe all those nice pictures we post
are just taken from a stock photograph website? And the answer is:
You don't need to.
## Inspect yourself
Instead of believing us, we invite you to visit us,
the [Data Center Light](/u/projects/data-center-light/) in the heart
of [Digital Glarus](/u/projects/digital-glarus/). We invite you to
visit, to take pictures, to convince yourself. To post the pictures
and to share it.
To have a look at the hydro power plant (its loud!), the solar panels
(luckily not loud) and the server infrastructure. You will not
even find any cooling mechanism, because we don't cool our servers.
## Visit and stay
If just visiting to find out whether or not our claim is true sounds
like wasting a lot of resources, then why don't you even stay with us
and work or relax?
We can provide you with a place to sleep and stay over in the
[Hacking Hotel](/u/projects/hacking-hotel/) or the
[Digital Chalet](/u/projects/digital-chalet/), where you can work with
a 10 Gbit/s connection that goes directly to the data center.
## Why?
Here at ungleich we believe that full transparency is required to
establish trust. This is also why we launched the
[Open Infrastructure
which describes our internal infrastructure. This is also why Data
Center Light is solely based on Open Source Software.
And on top of all of this, we think that Glarus is already worth a
visit just for enjoying the very pretty view. Glarus is a lot like a
fairy tale, just in reality.
## How to visit
Just reach out to us by email (info at,
[chat]( or
[twitter]( and we will share information
of how and when to best reach us with you.

@ -0,0 +1,88 @@
title: Colocation
_discoverable: yes
_hidden: no
subtitle: Your space in Data Center Light
feature1_title: Renewable Energy
feature1_text: Your hardware is fully powered by renewable energy.
feature2_title: Located in Switzerland
feature2_text: Our data center locations are based in Glarus in the
middle of Switzerland. Not only is it one of the safest places on
earth, it is also naturally cool.
feature3_title: Customisable
feature3_text: Your colocation can be fully customised to your
needs. Space, connectivity, IPv4 and IPv6 addresses can be freely configured.
At [Data Center Light](/u/projects/data-center-light) we don't use
traditional racks. Instead we give each server enough space (on
average 4m² surrounding area). This way we don't need to actively cool.
## How it works
If you want to bring your own hardware, you can select from the
following options:
## Space
For one server we provide you with 1m² of physical shelf space. We
call this our "base unit".
Your server can be 1U, 2U or even 4U high. Contact us for special sizes.
We expect your server to not create more warmth than 3000 BTU/h. If it
produces more heat, you will need to add more base units.
## Electricity
Your electricity usage is monitored and charged monthly. The price for
electricity is 0.2 CHF/kWh.
## Connectivity
We offer 3 levels of connectivity:
* 10 Mbit/s (burstable 1000 Mbit/s) - 15 CHF/month
* 100 Mbit/s (burstable 1000 Mbit/s) - 75 CHF/month
* 1000 Mbit/s - 400 CHF/month
10 and 100 Mbit/s are calculated on the 95% rule.
Most customers select the 100 Mbit/s option.
Bandwidth exceeding the booked tarif is charged at 2.5 CHF/Mbps.
You get a multi mode fiber cable (850nm transceivers) to your shelf.
## IP addresses
* Colocation includes a /48 IPv6 network for free
* IPv4 addresses are provided using NAT64 for 9 CHF/IPv4 address/month
## Setup Fee
The one time setup fee is 100 CHF. It includes configuring a VLAN
dedicated for you and adding routing to your own network.
## Sample calculations
* 1 base unit, 100 Mbit, /48 IPv6, no IPv4 = 50 + 75 = 125 CHF/month
* 2 base units, 100 Mbit, /48 IPv6, 2 IPv4 addresses = 2*50 + 75 + 2*9 = 193 CHF/month
* 4 base units, 100 Mbit, /48 IPv6, 40 IPv4 addresses = 4*50 + 75 + 40*9 = 635 CHF/month
* 4 base units, 1000 Mbit/s, /48 IPv6, no IPv4 = 4*50 + 400 = 600 CHF/month
## Fine print
All prices listed as usual without vat.

@ -0,0 +1,62 @@
_discoverable: no
_hidden: no
title: Django Hosting
subtitle: Hassle free Django hosting
feature1_title: Proven over years
feature1_text: Here at ungleich we do not only provide django hosting
as a service, but we also run Django instances and we provide
customers with Django based services.
feature2_title: Scalable
feature2_text: With our django hosting you can grow as needed. You
can start with a small sites, including development, testing or
staging systems and later grow to full production systems including
feature3_title: Sustainable & Carbon free
feature3_text: Because our Django hosting runs inside
[Data Center Light](/u/projects/data-center-light), you can run your
application with a good conscience and without emitting extra carbon.
## Development flow
The below diagram gives an overview of a typical development flow with
our Django hosting:
content1_image: djangohosting.svg
## How it works
## Components
Our Django hosting consists of 1-2 virtual machines and an optional
backup. Our customers often start with a development system
We strongly recommend to develop Django (like any other application)
with a version control system, even if you are just a single
developer. You can even use the [ungleich code
repository]( for free
(you only need to [register an account](
## Pricing
* Setup fee is 35 CHF
* Monthly fee is 20 CHF + price of the VM
* Typical development VM: 1 cores, 2 GB RAM, 10 GB SSD
* Typical small prod VM: 2 cores, 4 GB RAM, 20 GB SSD
* Typical default prod VM: 4 cores, 8 GB RAM, 50 GB SSD
* All VMs can be extended with extra HDD space

@ -0,0 +1,147 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
<!-- Generated by graphviz version 2.42.3 (0)
<!-- Title: G Pages: 1 -->
<svg width="760pt" height="443pt"
viewBox="0.00 0.00 760.00 443.00" xmlns="" xmlns:xlink="">
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 439)">
<polygon fill="white" stroke="transparent" points="-4,4 -4,-439 756,-439 756,4 -4,4"/>
<g id="clust1" class="cluster">
<polygon fill="none" stroke="black" points="8,-281 8,-427 400,-427 400,-281 8,-281"/>
<g id="clust2" class="cluster">
<polygon fill="none" stroke="black" points="109,-8 109,-248 642,-248 642,-8 109,-8"/>
<text text-anchor="middle" x="375.5" y="-232.8" font-family="Times,serif" font-size="14.00">First data center</text>
<g id="clust3" class="cluster">
<polygon fill="none" stroke="black" points="117,-16 117,-217 459,-217 459,-16 117,-16"/>
<text text-anchor="middle" x="288" y="-201.8" font-family="Times,serif" font-size="14.00">Production environment</text>
<g id="clust4" class="cluster">
<polygon fill="none" stroke="black" points="467,-17 467,-92 634,-92 634,-17 467,-17"/>
<text text-anchor="middle" x="550.5" y="-76.8" font-family="Times,serif" font-size="14.00">Staging environment</text>
<g id="clust5" class="cluster">
<polygon fill="none" stroke="black" points="408,-281 408,-356 708,-356 708,-281 408,-281"/>
<text text-anchor="middle" x="558" y="-340.8" font-family="Times,serif" font-size="14.00">Second data center</text>
<!-- editor -->
<g id="node1" class="node">
<polygon fill="none" stroke="black" points="247.5,-419 146.5,-419 146.5,-383 247.5,-383 247.5,-419"/>
<text text-anchor="middle" x="197" y="-397.3" font-family="Times,serif" font-size="14.00">Local editor</text>
<!-- git -->
<g id="node2" class="node">
<polygon fill="none" stroke="black" points="182,-325 16,-325 16,-289 182,-289 182,-325"/>
<text text-anchor="middle" x="99" y="-303.3" font-family="Times,serif" font-size="14.00">Local version control</text>
<!-- editor&#45;&gt;git -->
<g id="edge1" class="edge">
<path fill="none" stroke="black" d="M178.57,-382.7C163.37,-368.43 141.65,-348.04 124.74,-332.16"/>
<polygon fill="black" stroke="black" points="127.04,-329.52 117.35,-325.23 122.25,-334.62 127.04,-329.52"/>
<!-- localweb -->
<g id="node3" class="node">
<polygon fill="none" stroke="black" points="391.5,-325 200.5,-325 200.5,-289 391.5,-289 391.5,-325"/>
<text text-anchor="middle" x="296" y="-303.3" font-family="Times,serif" font-size="14.00">Development webserver</text>
<!-- editor&#45;&gt;localweb -->
<g id="edge3" class="edge">
<path fill="none" stroke="black" d="M215.62,-382.7C230.97,-368.43 252.91,-348.04 270,-332.16"/>
<polygon fill="black" stroke="black" points="272.52,-334.6 277.46,-325.23 267.75,-329.47 272.52,-334.6"/>
<!-- repo -->
<g id="node5" class="node">
<polygon fill="none" stroke="black" points="256.5,-186 125.5,-186 125.5,-148 256.5,-148 256.5,-186"/>
<text text-anchor="middle" x="191" y="-170.8" font-family="Times,serif" font-size="14.00">Code repository</text>
<text text-anchor="middle" x="191" y="-155.8" font-family="Times,serif" font-size="14.00">(f.i. git)</text>
<!-- git&#45;&gt;repo -->
<g id="edge2" class="edge">
<path fill="none" stroke="black" d="M110.47,-288.8C126.2,-265.2 154.64,-222.54 173.16,-194.75"/>
<polygon fill="black" stroke="black" points="176.17,-196.56 178.8,-186.3 170.34,-192.67 176.17,-196.56"/>
<!-- prodvm -->
<g id="node4" class="node">
<polygon fill="none" stroke="black" points="398,-62 228,-62 228,-24 398,-24 398,-62"/>
<text text-anchor="middle" x="313" y="-46.8" font-family="Times,serif" font-size="14.00">Production Server</text>
<text text-anchor="middle" x="313" y="-31.8" font-family="Times,serif" font-size="14.00">uwsgi,nginx,postgres</text>
<!-- backupvm -->
<g id="node8" class="node">
<polygon fill="none" stroke="black" points="699.5,-325 580.5,-325 580.5,-289 699.5,-289 699.5,-325"/>
<text text-anchor="middle" x="640" y="-303.3" font-family="Times,serif" font-size="14.00">Backup server</text>
<!-- prodvm&#45;&gt;backupvm -->
<g id="edge6" class="edge">
<path fill="none" stroke="black" d="M358.83,-62.1C388.13,-72.81 427.22,-85.51 463,-92 480.84,-95.24 613.06,-87.3 626,-100 673.73,-146.84 659.33,-234.8 648.01,-279.08"/>
<polygon fill="black" stroke="black" points="644.6,-278.28 645.38,-288.84 651.36,-280.1 644.6,-278.28"/>
<text text-anchor="middle" x="705.5" y="-163.3" font-family="Times,serif" font-size="14.00">Daily backup</text>
<!-- repo&#45;&gt;prodvm -->
<g id="edge4" class="edge">
<path fill="none" stroke="black" d="M167.56,-147.99C162.46,-142.8 157.77,-136.7 155,-130 149.9,-117.68 147.31,-110.89 155,-100 169.9,-78.91 193.45,-65.71 217.86,-57.47"/>
<polygon fill="black" stroke="black" points="219.1,-60.75 227.61,-54.44 217.03,-54.07 219.1,-60.75"/>
<text text-anchor="middle" x="228.5" y="-111.3" font-family="Times,serif" font-size="14.00">Deploy after staging</text>
<!-- stagingvm -->
<g id="node7" class="node">
<polygon fill="none" stroke="black" points="600.5,-61 475.5,-61 475.5,-25 600.5,-25 600.5,-61"/>
<text text-anchor="middle" x="538" y="-39.3" font-family="Times,serif" font-size="14.00">Staging Server</text>
<!-- repo&#45;&gt;stagingvm -->
<g id="edge5" class="edge">
<path fill="none" stroke="black" d="M256.74,-152.12C264.89,-150.62 273.12,-149.2 281,-148 356.55,-136.49 383.03,-162.91 452,-130 480.51,-116.4 505.08,-89.32 520.56,-69.16"/>
<polygon fill="black" stroke="black" points="523.47,-71.12 526.63,-61.01 517.85,-66.94 523.47,-71.12"/>
<text text-anchor="middle" x="567.5" y="-111.3" font-family="Times,serif" font-size="14.00">Pre prod deployment</text>
<!-- monitoring -->
<g id="node6" class="node">
<polygon fill="none" stroke="black" points="436,-185 290,-185 290,-149 436,-149 436,-185"/>
<text text-anchor="middle" x="363" y="-163.3" font-family="Times,serif" font-size="14.00">Monitoring server</text>
<!-- monitoring&#45;&gt;prodvm -->
<g id="edge7" class="edge">
<path fill="none" stroke="black" d="M330.58,-148.81C323.93,-143.69 317.81,-137.42 314,-130 304.96,-112.4 305.01,-89.98 307.16,-72.45"/>
<polygon fill="black" stroke="black" points="310.67,-72.66 308.71,-62.25 303.75,-71.61 310.67,-72.66"/>
<text text-anchor="middle" x="383" y="-118.8" font-family="Times,serif" font-size="14.00">Verify operation</text>
<text text-anchor="middle" x="383" y="-103.8" font-family="Times,serif" font-size="14.00">and alert on failure</text>
<!-- monitoringmonitor -->
<g id="node9" class="node">
<polygon fill="none" stroke="black" points="562,-325 416,-325 416,-289 562,-289 562,-325"/>
<text text-anchor="middle" x="489" y="-303.3" font-family="Times,serif" font-size="14.00">Monitoring server</text>
<!-- monitoringmonitor&#45;&gt;monitoring -->
<g id="edge8" class="edge">
<path fill="none" stroke="black" d="M427.45,-288.81C418.09,-284.17 409.18,-278.33 402,-271 381.78,-250.37 371.85,-218.21 367.1,-195.33"/>
<polygon fill="black" stroke="black" points="370.49,-194.42 365.21,-185.23 363.61,-195.7 370.49,-194.42"/>
<text text-anchor="middle" x="514" y="-259.8" font-family="Times,serif" font-size="14.00">Monitor the monitoring service</text>


Width:  |  Height:  |  Size: 7.9 KiB