[doc] workaround for local dns hairpinning failure #10

Closed
opened 2019-11-03 19:44:13 +00:00 by Ghost · 35 comments

Building a socket.io (sorry not webpush) with express app.
I have a custom greenlock.json for altnames/sites.

function httpsWorker(glx) {
	server = glx.httpsServer();
	glx.serveApp(app);
};
var glx = require("greenlock-express").init(function getConfig() {
		return {
			package: { name: "XXX", version:"E" },
			maintainerEmail: "XXXX@hotmail.com",  // *real email in code
			cluster: false,
			configFile : "greenlock.json",
			debug:true
		};
}).serve(httpsWorker);

Everything works fine until a client try to connect :

get adress info ENOTFOUND domain.fr 80. * where domain.fr is my real domain.

Building a socket.io (sorry not webpush) with express app. I have a custom greenlock.json for altnames/sites. function httpsWorker(glx) { server = glx.httpsServer(); glx.serveApp(app); }; var glx = require("greenlock-express").init(function getConfig() { return { package: { name: "XXX", version:"E" }, maintainerEmail: "XXXX@hotmail.com", // *real email in code cluster: false, configFile : "greenlock.json", debug:true }; }).serve(httpsWorker); Everything works fine until a client try to connect : get adress info ENOTFOUND domain.fr 80. * where domain.fr is my real domain.
Ghost changed title from [doc] to [doc] client cannot connect 2019-11-03 19:47:38 +00:00
Owner

That's a DNS resolution error.

Are you seeing that error on the server side, or on the client side?

Are you trying to run on localhost? (for that you need to use one of the DNS plugins with an API key or token from your DNS provider)

That's a DNS resolution error. Are you seeing that error on the server side, or on the client side? Are you trying to run on `localhost`? (for that you need to use one of the DNS plugins with an API key or token from your DNS provider)
Ghost closed this issue 2019-11-04 08:49:08 +00:00
Ghost reopened this issue 2019-11-04 08:49:19 +00:00
Author

Thanks for having a look. You are doing an awesome work!!
This is a server side error.
I am not on localhost but in local network. (loopback in my Synology router).
Please note that it is running well with current certificates and old let's encrypt script.

Morevoer, I am looking deeply in your code to try to make it works.
I don't understand the logic of having a file to configures the domains for greenlock-express. I don't see anyother way to configure it (I would like to have it as gl.add like for main library 'greenlock')

Thanks for having a look. You are doing an awesome work!! This is a server side error. I am not on localhost but in local network. (loopback in my Synology router). Please note that it is running well with current certificates and old let's encrypt script. Morevoer, I am looking deeply in your code to try to make it works. I don't understand the logic of having a file to configures the domains for greenlock-express. I don't see anyother way to configure it (I would like to have it as gl.add like for main library 'greenlock')
Owner

I would like to have it as gl.add like for main library greenlock'

That's the same as it was:

// See "JavaScript API" > "Greenlock#add" at
// https://git.rootprojects.org/root/greenlock.js

gl.add({ subject, altnames })

However, that's probably not what you want.
You probably want the CLI:

# (will be published to npm today)

npm install 'git+https://git.coolaj86.com/coolaj86/greenlock.js.git#next'
npx greenlock add --subject xxxx --altnames xxxx,xxxx

If you in fact do want the API from within Greenlock itself, I'd like to chat with you so that I can understand your use case and how to expose it.

I am not on localhost but in local network.

You have to have a public connection in order to do http-01 validations. You'll need to switch to dns-01 validation.

If you do have a public inbound connection... is the synology server local DNS misconfigured?

> I would like to have it as gl.add like for main library greenlock' That's the same as it was: ``` // See "JavaScript API" > "Greenlock#add" at // https://git.rootprojects.org/root/greenlock.js gl.add({ subject, altnames }) ``` However, that's probably not what you want. You probably want the CLI: ```bash # (will be published to npm today) npm install 'git+https://git.coolaj86.com/coolaj86/greenlock.js.git#next' ``` ```bash npx greenlock add --subject xxxx --altnames xxxx,xxxx ``` If you in fact do want the API from within Greenlock itself, I'd like to chat with you so that I can understand your use case and how to expose it. > I am not on localhost but in local network. You have to have a public connection in order to do http-01 validations. You'll need to switch to dns-01 validation. If you do have a public inbound connection... is the synology server local DNS misconfigured?
Author

Right ok but not for greenlock expres object. Maybe I misunderstood how to use greenlock-express. Sould I build a greenlock object before gl-express?

With your previous code letsencrpyt the http-01 challenge was working well to renew certificate.

It looks like
my port is blocked (I made the tip to open root port)...
or...REDIRECT HTTPP->HTPPS is wrong maybe?

Thank you for your time.
It is to run my own progressive web app I use for home automation. a personnal project.
It is quite demanding to have its own app for home automation due to maintenance and libraries update/depencies : I need to update my node.js >11.2 to run your code.

If I lost my certificate, I cannot benefit from advanced services i.e. notifications.
Alexandre.

Right ok but not for greenlock expres object. Maybe I misunderstood how to use greenlock-express. Sould I build a greenlock object before gl-express? With your previous code letsencrpyt the http-01 challenge was working well to renew certificate. It looks like my port is blocked (I made the tip to open root port)... or...REDIRECT HTTPP->HTPPS is wrong maybe? Thank you for your time. It is to run my own progressive web app I use for home automation. a personnal project. It is quite demanding to have its own app for home automation due to maintenance and libraries update/depencies : I need to update my node.js >11.2 to run your code. If I lost my certificate, I cannot benefit from advanced services i.e. notifications. Alexandre.
Owner

I need to update my node.js >11.2 to run your code.

It should be compatible with node v8+. I've been testing mostly on v10.

There a warnings about missing features in < 11.2, but it does not require an update.

With your previous code letsencrpyt the http-01 challenge was working well to renew certificate

Hmm... the challenge module is the same module.

greenlock.add()

Why not use the CLI? The .add() function is only intended for dynamic adding of domains via API. It is not intended for single domains.

I see the error in your code snippet, and I can give you the correction, but I want to understand: Why are you trying to do it this way?

Do you need the API for multiple, dynamic domains? or are you just adding a single domain?

ENOTFOUND

What happens when you do this, from your synology box?

dig YOUR_DOMAIN.fr
curl http://YOUR_DOMAIN.fr

Do dig and curl also give you an error?

> I need to update my node.js >11.2 to run your code. It should be compatible with node v8+. I've been testing mostly on v10. There a _warnings_ about missing features in < 11.2, but it does not _require_ an update. > With your previous code letsencrpyt the http-01 challenge was working well to renew certificate Hmm... the challenge module is the same module. > greenlock.add() Why not use the CLI? The `.add()` function is only intended for dynamic adding of domains via API. It is not intended for single domains. I see the error in your code snippet, and I can give you the correction, but I want to understand: Why are you trying to do it this way? Do you need the API for multiple, dynamic domains? or are you just adding a single domain? > `ENOTFOUND` What happens when you do this, from your synology box? ```bash dig YOUR_DOMAIN.fr ``` ```bash curl http://YOUR_DOMAIN.fr ``` Do `dig` and `curl` also give you an error?
Owner

Can you get on chat?

I'm @coolaj86 on Keybase.

Can you get on chat? I'm @coolaj86 on Keybase.
coolaj86 changed title from [doc] client cannot connect to [doc] workaround for local dns failure 2019-11-05 11:06:37 +00:00
Owner

I've updated the docs. Let me know what you think.

I've updated the docs. Let me know what you think.
Author

Nice to talk to you by phone/screen sharing yesterday!
When doing the command
sudo vi /etc/resolv.conf (*sudo otherwise read-only)

I changed to google dns servers instead of my router local address 192.168.1.1
nameserver 8.8.8.8
nameserver 8.8.4.4

below image is the new error.
the port 80 is already used. I will double check if don't launch 'twice' the same command on my side and come back later...but I doubt.

Nice to talk to you by phone/screen sharing yesterday! When doing the command sudo vi /etc/resolv.conf (*sudo otherwise read-only) I changed to google dns servers instead of my router local address 192.168.1.1 nameserver 8.8.8.8 nameserver 8.8.4.4 below image is the new error. the port 80 is already used. I will double check if don't launch 'twice' the same command on my side and come back later...but I doubt.
Owner

EADDRINUSE sometimes happens when a process exits while still listening, but there's definitely something else using port 80.

Try one of these:

lsof -i TCP:80

or

netstat --tcp --listening --programs --numeric
`EADDRINUSE` sometimes happens when a process exits while still listening, but there's definitely something else using port 80. Try one of these: ```bash lsof -i TCP:80 ``` or ```bash netstat --tcp --listening --programs --numeric ```
Author

After a reboot. A new error appears.
Fail to load greenlock-manager-fs.
I have tried unbuild / install without success.

After a reboot. A new error appears. Fail to load greenlock-manager-fs. I have tried unbuild / install without success.
Author

Result of your command when my app is running with node6 and old letsencrypts.

Result of your command when my app is running with node6 and old letsencrypts.
Owner

From the top

What node version? (should be v8+)

node --version
v11.14.0

Is there anything already listening on port 80?

sudo lsof -i TCP:80
COMMAND  PID USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
node    4277 root   21u  IPv4 5409381      0t0  TCP *:http (LISTEN)
sudo netstat --tcp --listening --programs --numeric
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      4277/node

Yes, there is. I have to kill it:

sudo killall node

Fresh, clean start in a new project directory:

mkdir -p my-project
pushd my-project
npm init -y
npm install --save greenlock-express@3

Initialize greenlock, fresh:

npx greenlock init
npx greenlock defaults --subscriber-email 'jon@example.com'
npx greenlock add --subject beta.example.org --altnames beta.example.org
npm start -- --staging

Get a certificate from the staging environment:

curl -k https://beta.rootprojects.org

Flawless

Check each one of those things and let me know what you get.

The unexpected token function sounds like node v6 encountering var x = async function () {};;

# From the top What node version? (should be v8+) ```bash node --version v11.14.0 ``` Is there anything already listening on port 80? ```bash sudo lsof -i TCP:80 ``` ``` COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME node 4277 root 21u IPv4 5409381 0t0 TCP *:http (LISTEN) ``` ```bash sudo netstat --tcp --listening --programs --numeric ``` ```bash Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 4277/node ``` Yes, there is. I have to kill it: ``` sudo killall node ``` Fresh, clean start in a new project directory: ```bash mkdir -p my-project pushd my-project npm init -y npm install --save greenlock-express@3 ``` Initialize greenlock, fresh: ```bash npx greenlock init npx greenlock defaults --subscriber-email 'jon@example.com' npx greenlock add --subject beta.example.org --altnames beta.example.org npm start -- --staging ``` Get a certificate from the staging environment: ```bash curl -k https://beta.rootprojects.org ``` # Flawless Check each one of those things and let me know what you get. The `unexpected token function` sounds like node v6 encountering `var x = async function () {};`;
Owner

You absolutely have another process listening on ports 80 and 443. The results of your netstat show that without question.

However, the output is truncated. You probably need to run it with sudo to see the process name.

You absolutely have another process listening on ports 80 and 443. The results of your netstat show that without question. However, the output is truncated. You probably need to run it with `sudo` to see the process name.
Author

I think I made some progress. but the /etc/resolv.conf came back to its previous. It is written on the top of the file "generated by resolvconf".
On my rasp PI 2, this file is managed by /etc/dhcpcd.conf. So I should modify this file instead. I wil lcome back.

I think I made some progress. but the /etc/resolv.conf came back to its previous. It is written on the top of the file "generated by resolvconf". On my rasp PI 2, this file is managed by /etc/dhcpcd.conf. So I should modify this file instead. I wil lcome back.
Author

I am back with a nodeJs 8.16.0.

  • I have difficulties to take control of /etc/resolv.conf that is changed by a network manager. I need to investigate.
  • new error appears : socket hang up. (si below)
I am back with a nodeJs 8.16.0. - I have difficulties to take control of /etc/resolv.conf that is changed by a network manager. I need to investigate. - new error appears : socket hang up. (si below)
Author

npx greenlock init
npx greenlock defaults --subscriber-email 'jon@example.com'
npx greenlock add --subject beta.example.org --altnames beta.example.org
npm start -- --staging

I failed at 'npm start --staging'.
Where to add the maintainerEmail?

npx greenlock init npx greenlock defaults --subscriber-email 'jon@example.com' npx greenlock add --subject beta.example.org --altnames beta.example.org npm start -- --staging I failed at 'npm start --staging'. Where to add the maintainerEmail?
Owner

Where to add the maintainerEmail?

In the author field of package.json, or in server.js (in init).

> Where to add the maintainerEmail? In the `author` field of `package.json`, or in `server.js` (in `init`).
Author
  • I found a way to keep control of dns resolv manager
  • I have found the origin of port 80 already occupied
  • I have updated greenlock-express (latest) and got a new error see image below. Looks like I absoltly need to move to node.js 11 minimum
- I found a way to keep control of dns resolv manager - I have found the origin of port 80 already occupied - I have updated greenlock-express (latest) and got a new error see image below. Looks like I absoltly need to move to node.js 11 minimum
Owner

If you set accountKeyType: "RSA-2048", it should get past that error. However... you'll be much better off with v10+.

I could put in a backport for ECDSA support on node v8. I think it's been there since v0.x, but it was exposed in a weird way until v10.x

If you set `accountKeyType: "RSA-2048"`, it should get past that error. However... you'll be much better off with v10+. I could put in a backport for ECDSA support on node v8. I think it's been there since v0.x, but it was exposed in a weird way until v10.x
Author

Hi.
I am back with nodejs v11.14.0. That was quite a bit of challenge to migrate serialport, openzwave and socket.io.
I just tested the app.
I have a nice message like everything is ok (see image below) but from chrome client nothing happens.

Please note that before (with old lex code) I created the server like that :

     var server = require('https').createServer(lex.httpsOptions, lex.middleware(app))

Now it is

            var httpsWorker = function (glx) {
	server = glx.httpsServer();
	glx.serveApp(app); };
Hi. I am back with nodejs v11.14.0. That was quite a bit of challenge to migrate serialport, openzwave and socket.io. I just tested the app. I have a nice message like everything is ok (see image below) but from chrome client nothing happens. Please note that before (with old lex code) I created the server like that : var server = require('https').createServer(lex.httpsOptions, lex.middleware(app)) Now it is var httpsWorker = function (glx) { server = glx.httpsServer(); glx.serveApp(app); };
Author

Error after some times...socket hang up

Error after some times...socket hang up
Owner

You've clipped it down only to the exact error message, so I can't tell what else is going on.

Did you add your domain to your config file?

You've clipped it down only to the exact error message, so I can't tell what else is going on. Did you add your domain to your config file?
Author

The full error message.
It is repeated itself several time.

The full error message. It is repeated itself several time.
Author

I have tried from a new project/directory as you explained the steps to make it work. but same results, nothing happens at client side.

I have tried from a new project/directory as you explained the steps to make it work. but same results, nothing happens at client side.
Author

for test project with npx greenlock init, everything looks good in greenlock.json.
I have tried again today. No progress. SOCKET HANG UP. I have open TCP 80 and 443 ports everywhere in my network.

for test project with npx greenlock init, everything looks good in greenlock.json. I have tried again today. No progress. SOCKET HANG UP. I have open TCP 80 and 443 ports everywhere in my network.
40 KiB
Author

Short questions concerning this local issuing of domains:

Does this mean I can also have a local offline https generation so that I can use the same server setup for testing, development and online (apart from the store or method being different?)

Or do I need to setup some server stuff and this is rather for container communication or something?

Thank you for clarifying.

Short questions concerning this local issuing of domains: Does this mean I can also have a local offline https generation so that I can use the same server setup for testing, development and online (apart from the store or method being different?) Or do I need to setup some server stuff and this is rather for container communication or something? Thank you for clarifying.
Owner

This is really a different question that you're asking, not related to this thread, but I can see how you might have mistaken it (local dns).

local offline https generation

Yes, but you need to use a DNS challenge. See the README.

this is rather for container communication or something

This thread is in regards to a DNS resolution failure. It's a highly technical and unusual setup in which the DNS server isn't a commonplace router you'd have in a normal home or office, but rather a full-blown dedicated DNS server, and one which happens to not handle "hairpinning" correctly (ideally it would resolve internal requests in an optimized way, but still respond to the same queries as if they were for external domains).

This is really a different question that you're asking, not related to this thread, but I can see how you might have mistaken it (local dns). > local offline https generation Yes, but you need to use a DNS challenge. See the README. > this is rather for container communication or something This thread is in regards to a DNS resolution failure. It's a highly technical and unusual setup in which the DNS server isn't a commonplace router you'd have in a normal home or office, but rather a full-blown dedicated DNS server, and one which happens to not handle "hairpinning" correctly (ideally it would resolve internal requests in an optimized way, but still respond to the same queries as if they were for external domains).
coolaj86 changed title from [doc] workaround for local dns failure to [doc] workaround for local dns hairpinning failure 2019-11-12 00:44:49 +00:00
Author

I was not aware to have a special setup. I just want to make my raspberry accessible from internal as well as external network. I use the DNS server app of my synology router to make a hairpinning/loopback (see config below). It works fine with my current code version

I was not aware to have a special setup. I just want to make my raspberry accessible from internal as well as external network. I use the DNS server app of my synology router to make a hairpinning/loopback (see config below). It works fine with my current code version
Owner

I don't see any config attached here.

The version that you had didn't do a self-test, hence the misconfigured hairpinning didn't cause a problem. I'll have a way to turn off the self-test as well.

Yeah, advanced DNS like what you have with the Synology is tricky beast. You won't have these kind of issues with your typical consumer router.

P.S. I'm not at NAT guru, so I really won't be able to help you with the DNS issues other than to say use an external resolver.

I once spent several hours with a networking expert trying to get hairpinning working one of these advanced routers and had no success.

I don't see any config attached here. The version that you had didn't do a self-test, hence the misconfigured hairpinning didn't cause a problem. I'll have a way to turn off the self-test as well. Yeah, advanced DNS like what you have with the Synology is tricky beast. You won't have these kind of issues with your typical consumer router. <small> P.S. I'm not at NAT guru, so I really won't be able to help you with the DNS issues other than to say use an external resolver. I once spent several hours with a networking expert trying to get hairpinning working one of these advanced routers and had no success. </small>
Author

Config attached.
My internet consumer router does not have the loopback option. This is why I use this tricky beast. Do you known a better way to have domain address pointing to local raspberry when client inside local network?

okay forget my previous question. How to remove local test? I can do a quick test before you release a code.

Config attached. My internet consumer router does not have the loopback option. This is why I use this tricky beast. Do you known a better way to have domain address pointing to local raspberry when client inside local network? okay forget my previous question. How to remove local test? I can do a quick test before you release a code.
Owner

Given my experience and my lack of proper understanding of NAT I would have the DNS server configured to give a different address to an internal request than an external request and skip the NAT issue altogether.

However, you'd perhaps still have to configure a firewall to do mangling of DNS requests, or you'd perhaps have to have to DNS servers.

Hmm...

I think this is what I'd do:

  1. Set up Static DHCP, so that the Pi always gets the same address
  2. Use external DNS for the domain (i.e. name.com)
  3. Use a local DNS for caching (i.e. dnsmasque)
  4. Edit the local DNS to always point to the static DNS for internal requests
Given my experience and my lack of proper understanding of NAT I would have the DNS server configured to give a different address to an internal request than an external request and skip the NAT issue altogether. However, you'd perhaps still have to configure a firewall to do mangling of DNS requests, or you'd perhaps have to have to DNS servers. Hmm... I think this is what I'd do: 1. Set up _Static_ DHCP, so that the Pi always gets the same address 2. Use external DNS for the domain (i.e. name.com) 3. Use a local DNS for caching (i.e. dnsmasque) 4. Edit the local DNS to always point to the static DNS for internal requests
Owner

The big question is WHY can't your raspberry pi access local DNS?

You CAN'T do curl http://yourdomain.tld from the Pi, we tested that.

Can you even do any DNS at all from the Pi? Can you do curl https://google.com?

The big question is WHY can't your raspberry pi access local DNS? You CAN'T do `curl http://yourdomain.tld` from the Pi, we tested that. Can you even do _any_ DNS at all from the Pi? Can you do `curl https://google.com`?
Author

With my current settings, app is on. Everything looks good. (see image) : http: redirecting ok, https: forwarded to login page.

With my current settings, app is on. Everything looks good. (see image) : http: redirecting ok, https: forwarded to login page.
Author

Today, the test project says "Hello encrypted wrold". Without changing the resolv.conf file!
I tried to replicate the test envrionement in my project folder.
Unfortunatly, "npx greenlock add" in cli return "greenlock command not found".
I have tried uninstall/unbuild greenlock /-express / root. and reinstall clean but no progress so far.

... Good news. It works fine...until I try to run the program on start (with a bash in reboot file). The greenlock.json is generated in root of raspberry with high privilege, and npx greelock add command ask again.

Today, the test project says "Hello encrypted wrold". Without changing the resolv.conf file! I tried to replicate the test envrionement in my project folder. Unfortunatly, "npx greenlock add" in cli return "greenlock command not found". I have tried uninstall/unbuild greenlock /-express / root. and reinstall clean but no progress so far. ... Good news. It works fine...until I try to run the program on start (with a bash in reboot file). The greenlock.json is generated in root of raspberry with high privilege, and npx greelock add command ask again.
Author

Last comment for this issue. It is now solved and in "production".
Lessons learned :

  • thanks to this issue my app is now upgraded from node6 to node11.
  • my production directory was show corrupted. not possible to execute in cli the command "glx greenlock init/add". So I had to create a new production directory and reinstall everything. my app is now just fresh and clean!
  • the cli command to create certificate are the things that works for me.
  • the code needs to be executed from the folder : forever /path/to/the/project.js will not make it work. You must make a cd "/path/to/the/" & forever project.js" to make it works in a bash file.
  • I don't need to change my resolv.conf file neither my loopback/hairpinning network settings.

=> Awesome work @Root !

Last comment for this issue. It is now solved and in "production". Lessons learned : * thanks to this issue my app is now upgraded from node6 to node11. * my production directory was show corrupted. not possible to execute in cli the command "glx greenlock init/add". So I had to create a new production directory and reinstall everything. my app is now just fresh and clean! * the cli command to create certificate are the things that works for me. * the code needs to be executed from the folder : forever /path/to/the/project.js will not make it work. You must make a cd "/path/to/the/" & forever project.js" to make it works in a bash file. * I don't need to change my resolv.conf file neither my loopback/hairpinning network settings. => Awesome work @Root !
Ghost closed this issue 2019-11-15 10:44:41 +00:00
Sign in to join this conversation.
No Label
No Milestone
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: root/greenlock-express.js#10
No description provided.