Delivery Frequent Issues

Sean Horn -

The most common causes of problems with Delivery jobs are the following

1. Delivery Chef Server and builders not time-synced.

All systems installed with Chef software that interact should have their times synced with NTP. Also see Why Do My Build Jobs Sit in "Waiting for worker." Status?

The above in more detail with the symptoms and possible fix is

Given: 

Seeing old (>20 min old) messages about bad signatures in push jobs log
Seeing timeouts in the delivery logs when attempting to start a job


Fix:

Ensure time is in-sync across clients and server
Stop All push-jobs clients on all builders
Note: you may need to `kill -9` the service
Restart push-jobs server on Delivery chef server
Note: you may need to `kill -9` the service
Start push-jobs clients

2. Proxies. Get rid of them. If you can't get rid of them, you need to cover as many software components as possible with one setting.

 
kevin [16:29] Hi all— the build nodes are not getting proxy settings when the build cookbook runs “berks” — is there a recommended way to set proxy settings on the build nodes? Currently we are setting them in `/etc/environment`. We run `berks` by hand and berks works; the automated build process fails with a timeout
 
afiune [16:31] You configure the proxy in the environment file inside the delivery-cluster repo.
 
kevin [16:33] @afiune: Ok, so those proxy settings should just go into the delivery-cluster/environments file?
So if we already set up a cluster without doing that, is there a good way to set the config file and reconfigure?
 
afiune [16:35] Yes! There and make sure you run the automation to apply the changes.. Another place you want to configure it is in the `/etc/profile` as environment variables and export them. Make sure root has it when you login to the servers.
Internally we dispatch the job using the `delivery-cmd` that will respect your environment variables

3. Github organization linking

 
galen [14:58] Does this issue sound familiar to anyone? 
We seem to be having some trouble using github for SCM for a project. When we try to create a new project using github, we get a 500 internal server error. We have set up delivery as an authorized application in github. Any idea what we could try?
 
jmink [15:18] So the enterprise is linked between github and delivery
and the Delivery UI makes it look like you could link, so jmink/foobar to github_integration/foobar, but that doesn't actually work.  You'd need to link jmink/foobar to jmink/foobar
 
The github organization the repo is under needs to match the delivery server organization.
 
 
3. Untested upgrades of Delivery software
 
Please test all upgrades on non-critical systems before doing an in-place upgrade in a system that is used to do work.
Have more questions? Submit a request

Comments

Powered by Zendesk