{"id":382,"date":"2017-04-21T09:32:00","date_gmt":"2017-04-21T09:32:00","guid":{"rendered":"https:\/\/questy.org\/?p=382"},"modified":"2024-09-26T19:03:02","modified_gmt":"2024-09-26T19:03:02","slug":"scaling-puppet-enterprise-part-iiia-additional-compilers","status":"publish","type":"post","link":"https:\/\/questy.org\/index.php\/2017\/04\/21\/scaling-puppet-enterprise-part-iiia-additional-compilers\/","title":{"rendered":"Scaling Puppet Enterprise &#8211; Part IIIa &#8211; Additional Compilers"},"content":{"rendered":"\n<p><strong><em>You should have completed a split install before beginning this section. You can find the Split Installation documentation at Puppet&#8217;s Website, or the first installment of this tutorial <a href=\"http:\/\/questy.org\/blog\/2017\/04\/18\/scaling-puppet-enterprise-part-ii-installation\/\">here<\/a>. If you try and begin here, you might find yourself lost.<\/em><\/strong><\/p>\n\n\n\n<p><strong><em>Note also that the &#8220;Additional Compilers&#8221; docs comes in two parts&#8211;One to install the Load Balancer and one to install the compilers.<\/em><\/strong><\/p>\n\n\n\n<h3 class=\"wp-block-heading\">First, Some Philosophy<\/h3>\n\n\n\n<p>The Puppet Enterprise documentation circa PE 2015.3.2 had some &#8220;issues&#8221;. Let me actually preface that, though. Puppet Labs&#8217; documentation is by far some of the most voluminous and in many respects most complete vendor documentation out there. I don&#8217;t mean to disparage their work AT ALL. When it comes to the fact they even have documentation at this level, they&#8217;re the &#8220;bees knees&#8221;.<\/p>\n\n\n\n<p>However, I&#8217;ve always written documentation to fit the &#8220;grandma rule&#8221;. My grandmother was a little 4 foot nothing Cajun woman with English as her second language. She never used the first computer, still had a rotary phone when she passed away, and remained suspicious of anything technical. She <em>was<\/em>, however, a voracious reader, keenly intelligent, and understood considerably more than you&#8217;d expect on first glance. She also was a stickler for puncutation, grammar, and the like. In short, if my grandma couldn&#8217;t read the documentation and follow a step-by-step process to install Puppet successfully, then its just either too complex, poorly formatted or unclear and needs to be simplified.<\/p>\n\n\n\n<p>This causes a problem, of course. There are technologists out there that would become annoyed at repetition, verbosity around &#8220;understood&#8221; things, and spelling out each and every step along the way\u2026 even painfully. However, I feel it is the only <em>proper<\/em> way to document something. My rules are simple.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Leave nothing to question<\/li>\n\n\n\n<li>Be as verbose and clear as possible<\/li>\n\n\n\n<li>Make sure everything is in order, step-by-step<\/li>\n<\/ul>\n\n\n\n<p>By following this simple guideline, I feel I&#8217;m doing more of a service to the reader than if I presumed on their level of sophistication with Puppet, Linux\/UNIX, Windows, research capability, Google-foo or whatever.<\/p>\n\n\n\n<p>So let&#8217;s dive in, shall we?<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">HAProxy<\/h2>\n\n\n\n<p>Seemingly counterintuitive, now that we&#8217;ve done a split install, I want to next install the HAProxy we will use as a Load Balancer on the additional compilers. By installing this first, we can utilize Puppet to install the HAProxy, and manage them automatically rather than doing a lot of ad-hoc work.<\/p>\n\n\n\n<p>Also, by doing the proxy first, the prerequisites are satisfied in their proper order, the Load Balancer exists before configuring additional compilers (to be able to utilize the dns_alt_names for the load balancer along with the compilers) and to have the GitLab in place and hosting the control_repo before turning on and configuring Code Manager.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Hardware<\/h3>\n\n\n\n<p>In the initial hardware list, I included a node called &#8220;Compile Master&#8221;. This node looked like:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img data-recalc-dims=\"1\" decoding=\"async\" src=\"https:\/\/i0.wp.com\/ssconsultinggroup.net\/wp-content\/uploads\/2020\/05\/compile_master_specs.png?w=900\" alt=\"\" class=\"wp-image-1523\"\/><\/figure>\n\n\n\n<p>This node may seem like overkill, but disk and memory are cheap. If you are scaling at this level, its better to not have to reinstall your Load balancer later. Keep in mind, you don&#8217;t have to use HAProxy and can use a corporate Load Balancer here, but its configuration is outside the scope of this tutorial.<\/p>\n\n\n\n<p>Once you&#8217;ve provisioned the load balancer, ssh to the node as the root user, and use the &#8220;frictionless installer&#8221; to add your Puppet agent.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>curl -k https:\/\/master.example.com:8140\/packages\/current\/install.bash | bash<\/code><\/pre>\n\n\n\n<p>When the client is fully installed, retrieve the Enterprise Console from your browser, and navigate to Nodes | Classification | Unsigned Certificates and select &#8220;Accept All&#8221;. Finally, ssh to the instance as the root user and run <strong><em>puppet agent -t<\/em><\/strong> to finish the setup.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Configure the Load Balancer<\/h2>\n\n\n\n<p>At this point, the node is provisioned and you have a Puppet agent running on it, but you have as of yet not configured the HAProxy Load Balancer for use in the environment. The load balancer will be necessary to have in place prior to adding compile masters to your existing split installation. The following instructions guide you through setting up the HAProxy load balancer.<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>SSH to the Puppet Master as root. <em>(<strong>master.example.com<\/strong> in our list)<\/em><\/li>\n\n\n\n<li>Install the HAPRoxy Forge Module on the master<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>puppet module install puppetlabs-haproxy<\/code><\/pre>\n\n\n\n<p><br><br><em>leave your root console open while performing steps 3-6<\/em><\/p>\n\n\n\n<ol start=\"3\" class=\"wp-block-list\">\n<li>Retrieve the Enterprise Console in your browser<\/li>\n\n\n\n<li>Select <strong>Nodes<\/strong> | <strong>Classification<\/strong><\/li>\n\n\n\n<li>Create a New Classification Group called &#8220;<strong>Load Balancer<\/strong>&#8220;<\/li>\n\n\n\n<li>Select the new group from the list and pin the node &#8220;<strong>compiler.example.com<\/strong>&#8221; into the new group.<\/li>\n\n\n\n<li>In your open SSH session to <strong>master.example.com<\/strong>, create the profiles module to hold the configuration for HAProxy<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>cd \/etc\/puppetlabs\/code\/environments\/production\/modules\n\nmkdir -p profiles\/manifests\n\ncd profiles\/manifests<\/code><\/pre>\n\n\n\n<ol start=\"8\" class=\"wp-block-list\">\n<li>Once you have changed to the profiles\/manifests directory, create the loadbalancer.pp manifest.<\/li>\n\n\n\n<li>Follow the documentation <a href=\"https:\/\/forge.puppet.com\/puppetlabs\/haproxy\/readme\">here<\/a> to configure HAProxy. When complete, the loadbalancer.pp manifest should resemble the following with IPs corrected for your particular instance:<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code># Load Balancer Profile\nclass profiles::loadbalancer {\n\n  class { 'haproxy': }\n\n  # Main Proxy Listener\n  haproxy::listen { 'compiler.example.com':\n    collect_exported =&gt; false,\n    ipaddress        =&gt; $::ipaddress,\n    ports            =&gt; '8140',\n  }\n\n  # First Load balanced Compile Master\n  haproxy::balancermember { 'compiler1.example.com':\n    listening_service =&gt; 'compiler.example.com',\n    server_names      =&gt; 'compiler1.example.com',\n    ipaddress         =&gt; '10.0.1.24',\n    ports             =&gt; '8140',\n    options           =&gt; 'check',\n  }\n\n  # Second Load Balanced Compile Master\n  haproxy::balancermember { 'compiler2.example.com':\n    listening_service =&gt; 'compiler.example.com',\n    server_names      =&gt; 'compiler2.example.com',\n    ipaddress         =&gt; '10.0.1.25',\n    ports             =&gt; '8140',\n    options           =&gt; 'check',\n  }\n}<\/code><\/pre>\n\n\n\n<p>Once you have created this profile, retrieve the Puppet Enterprise Console in your browser and navigate to <strong>Nodes | Classification | Load Balancer<\/strong>.<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Selet the <strong>Classes<\/strong> tab.<\/li>\n\n\n\n<li>Click the &#8220;refresh&#8221; button so the console will pick up your new loadbalancer.pp profile to classify your node with.<\/li>\n\n\n\n<li>Under the &#8220;Add new Class&#8221; heading, select <strong>profiles::loadbalancer<\/strong> from the list that drops down.<\/li>\n\n\n\n<li>Click &#8220;Add Class&#8221;.<\/li>\n\n\n\n<li>Select &#8220;Commit 1 Change&#8221; at the bottom right of the page.<\/li>\n\n\n\n<li>SSH back into <strong>compiler.example.com<\/strong> and run <strong>puppet agent -t<\/strong> to configure the Load Balancer.<\/li>\n<\/ol>\n\n\n\n<p>Your Load Balancer is now prepared to balance traffic to two catalog compilers (<em><strong>catalog1.example.com<\/strong> and <strong>catalog2.example.com<\/strong><\/em>) as listed in the above configuration.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">###Notes<\/h2>\n\n\n\n<p>I noted when putting together the loadbalancer.pp profile above that I had previously used some REALLY ODD ip addresses in the balancer config. Why? For the life of me I cannot recall. The original file looked like so:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># Load Balancer Profile\nclass profiles::loadbalancer {\n\n  class { 'haproxy': }\n\n  # Main Proxy Listener\n  haproxy::listen { 'compiler.example.com':\n    collect_exported =&gt; false,\n    ipaddress        =&gt; $::ipaddress,\n    ports            =&gt; '8140',\n  }\n\n  # First Load balanced Compile Master\n  haproxy::balancermember { 'compiler1.example.com':\n    listening_service =&gt; 'compiler.example.com',\n    server_names      =&gt; 'compiler1.example.com',\n    ipaddress         =&gt; '10.0.1.24',\n    ports             =&gt; '8140',\n    options           =&gt; 'check',\n  }\n\n  # Second Load Balanced Compile Master\n  haproxy::balancermember { 'compiler2.example.com':\n    listening_service =&gt; 'compiler.example.com',\n    server_names      =&gt; 'compiler2.example.com',\n    ipaddress         =&gt; '10.0.1.25',\n    ports             =&gt; '8140',\n    options           =&gt; 'check',\n  }\n}<\/code><\/pre>\n\n\n\n<p>In my original implementation I set the ipaddres fields with some odd IP addresses. For info around how to fill those but ,the documentation gives some hints:<\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p>ipaddresses: Optional. Specifies the IP address used to contact the balancermember service. Valid options: a string or an array. If you pass an array, it must contain the same number of elements as the array you pass to the server_names parameter. For each pair of entries in the ipaddresses and server_names arrays, Puppet creates server entries in haproxy.cfg targeting each port specified in the ports parameter. Default: the value of the $::ipaddress fact.<\/p>\n<\/blockquote>\n\n\n\n<p>Since I was originally setting these up in Digital Ocean, I used the IP space 159.203.x.x which belongs to Digital Ocean. I am guessing these were the hard IPs on the instances I stood up. Since the documentation above states these are optional, you have two options here. Either leave those lines out of your config altogether, or manually set them to the IP Address of the instance you&#8217;re using. Try each and do which works for you.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Conclusion<\/h2>\n\n\n\n<p>Your HAProxy Load balancer is now complete and ready to take traffic to the additional catalog compiler nodes. In installment IV, we&#8217;ll begin to add in more components along the way to a fully developed LEI of Puppet Enterprise.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>You should have completed a split install before beginning this section. You can find the Split Installation documentation at Puppet&#8217;s Website, or the first installment of this tutorial here. If you try and begin here, you might find yourself lost. Note also that the &#8220;Additional Compilers&#8221; docs comes in two parts&#8211;One to install the Load [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"advanced_seo_description":"","jetpack_seo_html_title":"","jetpack_seo_noindex":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","enabled":false},"version":2}},"categories":[2,4],"tags":[],"class_list":["post-382","post","type-post","status-publish","format-standard","hentry","category-puppet-administration","category-puppet-education"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_likes_enabled":true,"jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/questy.org\/index.php\/wp-json\/wp\/v2\/posts\/382","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/questy.org\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/questy.org\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/questy.org\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/questy.org\/index.php\/wp-json\/wp\/v2\/comments?post=382"}],"version-history":[{"count":1,"href":"https:\/\/questy.org\/index.php\/wp-json\/wp\/v2\/posts\/382\/revisions"}],"predecessor-version":[{"id":383,"href":"https:\/\/questy.org\/index.php\/wp-json\/wp\/v2\/posts\/382\/revisions\/383"}],"wp:attachment":[{"href":"https:\/\/questy.org\/index.php\/wp-json\/wp\/v2\/media?parent=382"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/questy.org\/index.php\/wp-json\/wp\/v2\/categories?post=382"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/questy.org\/index.php\/wp-json\/wp\/v2\/tags?post=382"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}