title: How i run my ceph cluster or how you can make a killer storage solutions for almost free
---
pub_date: 2020-01-08
---
author: ungleich
---
twitter_handle: ungleich
---
_hidden: yes
---
_discoverable: no
---
abstract:
I wanted to store some data data and this is what I've came up with.
---

body:

Low cost, high tech datastorage with ceph
First of all why would you run a ceph cluster? Its complex, a bit time consuming and easier to lose data than using zfs or ext4.
My reasons:
its very easy to expand/shrink
manage all your data/disks from 1 host (can be a security risk too)
its fun
we have it in production and it scales well
unifying the physical 

Step 1 :
Find your local hw dealer. 
Second hand sites can be a good source, but good deals are rare. My tactics on ricardo.ch is: server & zubehor, filters: used, auction, max 1,5kchf, sorted with the ending soonest first) link : https://www.ricardo.ch/de/c/server-und-zubehoer-39328/?range_filters.price.max=1527&item_condition=used&offer_type=auction&sort=close_to_end
Nearby dangerous material (ewaste) handler companies can be a goldmine. Big companies cannot just throw used hardware out as regular waste becuse electronics contains a little amount of lead (or some other heavy metal). So big compnies sometimes happy to sell it as a used product for cheap and ewaste companies are happy if they get more money than the recycled price / kg which is very-very low

low quality (core2duo era) pc-s also suffice, but you wont be able to do erasure coded pools are they use a ton of processing power and ram.. be careful with the ram, if you run out of swap/ram, your osd process will be killed, learnt it the hard way. also sometimes recovery uses more ram than usual so keep some free ram for safety.
Put 10G nics on your shopping list, for performance, its absolutely crucial. I've started without it, and its certainly doable but it wont perform well. A little hack is to pick up gigabit nic cards (as some people give them away for free), and put them in an lacp bond. Note here: lacp doesnt make a single connection's speed better, the benefit is only realized at parallel connections.
If you dont care or have equal disks by size or speed no worries, ceph will happily consume everything you feed to it (except smr disks* or strictly only for frozen data) One hack is to snag some old/low capacity disks for free. If you do everyting right you can surpass ssd speeds with crappy spinning rusts. Worried about disks dying? Just have higher redundancy levels (keep 2 extra copies of your data) 
My personal approach is to have coldish data 2 times, hot data like vms 3x and 1 extra copy of both on non-ceph filesystem.
You can also group disks by performance/size. Ideally the disks should be uniform in a ceph device class, and equally distributed between hosts.
Avoid hardware raid, use cards that allow full control for to os over the disks. If you must use hw raid, raid0 is the way.

Install:
You can check out my ugly install script that meant to bootstrap a cluster on a vm.
tested on an alpine vm with an attached /dev/sdb datablock (don't use legacy ip (ipv4))

apk add bash
wget http://llnu.ml/data/ceph-setup
bash ./ceph-setup $ip_address_of_the_machine $subnet_that_you_will_plan_to_use

Operation:
I've never prepared a disks manually yet which i should definetly review, because Nico wrote amazing helper scripts which can be found in our repo: https://code.ungleich.ch/ungleich-public/ungleich-tools.git
Some scripts still need minor modifications because alpine doesnt ship ceph init scripts yet. For the time being I manage the processes by hand.

Alpine's vanilla kernel doesnt have rbd support compiled in it atm, but for any kernel that doesnt have the kernel module, just use rbd-nbd to map block devices from you cluster.




* https://blog.widodh.nl/2017/02/do-not-use-smr-disks-with-ceph/

Some useful commands:




Today we’re at Leipzig, Germany. Last day of [36c3](https://events.ccc.de/category/congress/36c3/), Chaos Communication Congress.

![](/u/image/xr-green.jpg)

## The earth is getting hotter

