Xcode 10 is now integrated with GitLab
twitter.com
twitter.com
I recently crossed swords with .gitlab-ci.yml. An obfuscation layer that fights back.
Ditto jenkinsfile and whatever you wanna call GoCD's config.xml.
This silliness reminds me most of all those ETL workflow obfuscation frameworks, like BizTalk and TalenD.
PS- If anyone has the bad judgement to use YAML, it'd be nice if the linter reported something more specific than "syntax error line 1 char 1" for deeply buried typos. I thought XML sucked until I met JSON. I thought JSON sucked until I met YAML. What's next? The resurrection of ASN.1?
PS2- It's not programming if you can't set a break point. The edit, eval, fail, profanity loop for troubleshooting .gitlab-ci.yml is pretty intense.
You still need something to describe the order and dependencies of those shell scripts as well as when to run them (branches, tags) and how to handle the artifacts (expiry time, etc). Which should be, much preferably, something not Turing-complete, so you can make an UI for this. Draw the graph, etc etc.
Take those parts out, and almost all what's left in .gitlab-ci.yml is just the `script` key.
You can debug most things locally (without having to do that ugly commit-try-fail-repeat dance) with https://docs.gitlab.com/runner/commands/#gitlab-runner-exec
If your visual programming fuzzy whatzzit drops down to metal (shell commands in this case), just use the metal.
I did get the git-runner to run locally. That was fun. Why do I have to commit my changes to my local branch so the git-runner can see them? MY CODE IS RIGHT THERE!
I reiterate: Just use shell scripts.
You'll need to run the CI pipeline to figure out the execution graph. In a restricted language, you don't have to - it can just describe that graph.
> If your visual programming fuzzy whatzzit drops down to metal
I don't get you. Everything programming eventually drops down to metal one way or another. It is not about visual programming, it's about the appropriate languages. Shell scripts are not exactly appropriate to describe a graph. Especially a graph meant to be ran on a distributed system - so, really, *sh may be not the best fit here. Maybe you have some better idea, but I'm just afraid you'll end up with some shell-based DSL that would be no better than YAML, but suffer from all possible issues (for example, there's no concept of pure functions in shell scripting, so every CI run plan can take completely different shape)
Oh. If you're a person who likes all their configuration files being programs - then I won't argue as it'll be a matter of taste.
> Why do I have to commit my changes to my local branch so the git-runner can see them?
Yeah, I'm also disappointed in this. It was dismissed as "wontfix" a while ago https://gitlab.com/gitlab-org/gitlab-runner/issues/1545 and seems that it stays this way even now (they've deprecated `exec` at some point but then reconsidered).
Had to write myself an alias for `git commit --amend && gitlab-runner exec`
"Everything programming eventually drops down to metal one way or another."
Keeping with the trog theme...
I need things to be as explicit as possible.
I just couldn't figure out why my script commands were not doing variable substitution. I expected my line ' - echo id: $CI_BUILD_ID' to cut and paste 'echo id: $CI_BUILD_ID' to the command line and run that bad boy as-is. And I'd see the expected output in the log. Just like if I run 'echo id: $CI_BUILD_ID' locally.
What could be more simple?
Nope.
gitlab-ci does some ninja code escape mangling thing (turn on debugging output to see it in action). So in your log you continue to see 'echo id: $CI_BUILD_ID' instead of 'id: 123456'.
I have no idea why anyone would do that.
I want to run commands. Why not just run them as-is? Why the obfuscation layers?
--
Inevitably, abstraction layers introduce an impedance mismatch, where some behavior will surprise you, where you have to do a workaround. Then you're fighting the tool instead of solving your problem directly. Then you say "fuck it" and just have the layer call your loose code (eg shell script) directly. Then what's the point of having the layer at all?
Of course. You may have different requirements, but here's an example flow of the one project I had worked on:
Lint --------------------
\
Build -+-- Tests (fast) --+-- Release --- Deploy
\ /
- Tests (slow) -
Lint runs in parallel with build + test (multiple runners, on different machines, each having some concurrency). Different tests run in parallel so you have to wait a little less. Tests only run after build is ready. Release only runs if both lint and tests succeed. Deploy is after the release.
You can write this graph as a shell script, but this is not without downsides:
1. You need a DSL (a bunch of pre-defined shell functions or external commands) that control the individual jobs in a sane manner.
2. You can't just look at the script and draw the pipeline graph (so before the pipeline's ran CI/CD can't tell you what's going to happen next).
3. You can't guarantee that the plan will be the same if pipeline is re-ran again. With shell scripts, program output may change, even though all inputs are the same.
If all you need is to run a simple shell script, with no multi-machine parallelism and no fancy conditional logic - all of the YAML stuff is an overkill you don't need. Although I guess all you need is approximately this:
run-ci:
script:
- ./run-ci.sh
> gitlab-ci does some ninja code escape mangling thing (turn on debugging output to see it in action). So in your log you continue to see 'echo id: $CI_BUILD_ID' instead of 'id: 123456'.
Not sure I've encountered this much, but yes, this probably makes things complex. I agree that this is can be a pain point.
> I have no idea why anyone would do that.
I'm not sure, but the probable rationale is likely that they didn't want secrets to leak. Variables are the place where you can store various secret information, like CI signing keys or remote server credentials. They probably didn't want to leak what happens with e.g. `curl --user "ci:$CI_DEPLOY_PASSWORD" "$CI_DEPLOY_URL"`.
I personally don't like to have a whole shell script inside .gitlab-ci.yaml file either -- that's why I have a seperate script called setup.bash/build.bash inside a CI folder and juse write `script: ./ci/build.bash` inside gitlab-ci.yaml. But nobody ever told you, you cannot do that. The gitlab-ci.yml is just a configuration file for the build system to tell it what you want your environment(s) to be.
I agree that error reporting for YAML files suck -- but at least they are more consistent and easier to parse then XML files. And they are also way more readable then JSON files -- they have essential features such as long strings, and comments. The only annoying thing with Gitlab CI (and any other CI system) is to make sure it works. It often takes me 10 tries to make sure everything works correctly.
"just write `script: ./ci/build.bash` inside gitlab-ci.yaml"
Yup. Then why have the YAML?
I said shell script, not bash. The first "CI/CD" (as the kids now call it) I ever did was on Windows. Worked great.
How would you simplify testing on multiple systems/oses? Or have one project written be tested for multiple versions of that language (because you need backwards compatibility)? It's nice if your build system is as simple as
make build test publish
I used to create prepress (print manufacturing) software for Mac (old and new) & Windows. Multi platform, multi executables, multi language, multi target builds. It'd pull all the resource bundles (L10N/I18N), recreate the the screen shots, rebuild the PDFs (manuals, marcom). The QA box farm would rerun all the regressions (combo of VirtualPC and VMware boxes) on multiple versions of multiple OSes (eg Japanese Windows) with all the necessary drivers (eg security dongles, which totally sucked). We'd end up with CD images and downloadable zips on our private website for VARs and high end customers.
Seemed to work pretty good.
I used to do a lot of electronic medical records stuff. My team (of 4 core devs) supported five regional exchanges (the first to market), 80 hospitals plus all their partners, pharma scripts, labs (internal and external), feeds to the govt (eg CDC, SSI). Basically 100s of data feeds of every mutant format and protocol imaginable. At the time, all of the data had to be stored on each customer's servers. Meaning we had to deploy and support in situ. Meaning firewalls, reverse SSH, forklifting files into place, whatever was permitted. We did multiple builds per week. Most deploys were push. Some we had to setup as pulls. We implemented auto updates, but never had the QA/Test resources to achieve the necessary confidence, which was a shame.
Seemed to work pretty good.
Sure, today's cloud deploys are complicated in other ways. But the more stuff changes, the more it all looks the same.
PS- Rereading your comment... Maybe consider changing some of the steps from push to pull. I was super inspired by the architecture of postfix (email server). Trying to mimick it simplified a lot of my own efforts.
It’s not very useful though. It just lets you clone repos without using a shell. Kind of pointless.
As others have said, support for two platforms was added. And GitHub was already supported. Guess that doesn’t make a great headline though.
I believe Amazon has something similar.
Does Google also have an issue tracker?
Heck, Chrome and Android both use (separate) in-house ones.
As for issue trackers, there is the chromium one that is based off of the old Google code issue tracker [1]. And about a year ago they started exposing a newer issue tracker[2].
[0] https://cloud.google.com/container-builder/
According to the 2017 IPO Report [1], citing SEC data, the median IPO offering size in 2016 was $95 million, and the median annual revenue was $66.5 million. To pick an example according to Crunchbase, Mulesoft raised $221 million in its 2017 IPO at a valuation of about $4 billion. They had a revenue in 2016 of $188 million with a year-over-year growth rate of 70%. For another example, Dropbox's 2018 IPO valued the company at $11 billion on revenues in 2017 of $1.1 billion.
[1] https://corpgov.law.harvard.edu/2017/05/25/2017-ipo-report/
However I'm already familiar with the topic. I'd just like to hear from Gitlab's CEO what makes him believe he's "on track" to an IPO in 2020. Usually such a display of confidence is backed by some form of evidence. Does Gitlab publish their revenue, or revenue growth, anywhere?
Kidding asside is any one surprised? With the way the Vally rumor mill works its shocking when things manage to stay secret.
I don't know if i'd go so far as to call it a tacit endorsement of gitlab by Apple, but it does have an air of opportunism.