Push Notifications stopped working

[ssgsgugel]

Issue Description

We have been running parse-server on AWS hosted ubuntu servers for a few months now and have been successfully pushing notifications out through the cloud push features for our iOS app. At some point between February 6th and February 13th, our push notifications stopped working through Parse and APN. As shown in the sample log statement below, we receive a success back from the send; however, our dashboard indicates the status is stuck at "SENDING". No errors are reported. We have updated the .p12 file with the latest generated from our developer site. However, no luck in getting it to work.

We have scanned both parse, apple, and AWS support areas but haven't see any issues reported.

Push Configuration

push: {
ios: {
pfx: 'certs/########.p12',
bundleId: 'io.#####.#####',
production: true
}
}

The master key is also specified in the server code.

Sample of our cloud main.js file that has been working fine.

Parse.Push.send({
where: pushQuery,
data: {
alert: message,
badge: "Increment",
sound: 'default',
objectId: matchId
}
},{
useMasterKey: true,
success: function() {
console.log('Message sent successfully');
res.success('Message was sent successfully');
},
error: function(error) {
console.log('Error sending push');
res.error(error);
}
});

Environment Setup

• Server

  • parse-server version (Be specific! Don't say 'latest'.) : Express 1.4.0
  • Operating System: Ubuntu 14.04.5 LTS (GNU/Linux 3.13.0-105-generic x86_64)
  • Hardware: AWS ECS
  • Localhost or remote server? (AWS, Heroku, Azure, Digital Ocean, etc): AWS

• Database

  • MongoDB version: v3.4.15
  • Storage engine:
  • Hardware: Same as above
  • Localhost or remote server? (AWS, mLab, ObjectRocket, Digital Ocean, etc): AWS

Logs/Trace

{"functionName":"sendpush","params":{"matchId":"####","message":"Test","senderId":"######","recipName":"#####","recipId":"######"},"user":"######","level":"info","message":"Ran cloud function sendpush for user ###### with:\n Input: {"matchId":"#######","message":"Test","senderId":"#######","recipName":"####","recipId":"######"}\n Result: "Message was sent successfully"","timestamp":"2019-02-16T22:29:44.110Z"}

Note: All ids and sensitive information was replaced with "#####"

[ssgsgugel]

In addition, our dev environment is running an Elastic Beanstalk version and we are experiencing similar issue there, so different version of Parse-server, but same issue.

[mtrezza]

Enable push adapter verbose logging with VERBOSE_PARSE_SERVER_PUSH_ADAPTER=1 to see more details. Also check if the APNS certificate has expired.

[ssgsgugel]

Will enable verbose logging; however, I can tell you our APNS certificate has not expired.

[ssgsgugel]

Okay, I think we have the log level set-up correctly. Once again, I attempted too hide some of the user specific information. Here were sections of the log at the time push notifications attempted to send:

{"method":"POST","url":"/dateset/push","headers":{"user-agent":"node-XMLHttpRequest, Parse/js1.9.2 (NodeJS 5.12.0)","accept":"/","content-type":"text/plain","host":"ec2-##########.compute-1.amazonaws.com:5656","content-length":"340","connection":"close"},"body":{"data":{"alert":"Matt: Test post upgrade","badge":"Increment","sound":"default","objectId":"8QZWnR0lZb"},"where":{"user":{"__type":"Pointer","className":"_User","objectId":"##########"}}},"level":"verbose","message":"REQUEST for [POST] /dateset/push: {\n "data": {\n "alert": "Matt: Test post upgrade",\n "badge": "Increment",\n "sound": "default",\n "objectId": \”########\”\n },\n "where": {\n "user": {\n "__type": "Pointer",\n "className": "_User",\n "objectId": \”########\”\n }\n }\n}","timestamp":"2019-02-28T14:09:19.758Z"}
{"result":{"headers":{"X-Parse-Push-Status-Id":"8ciuPXTRqe"},"response":{"result":true}},"level":"verbose","message":"RESPONSE from [POST] /dateset/push: {\n "headers": {\n "X-Parse-Push-Status-Id": "8ciuPXTRqe"\n },\n "response": {\n "result": true\n }\n}","timestamp":"2019-02-28T14:09:19.765Z"}
{"functionName":"sendpush","params":{"matchId”:”#######”,”message":"Test post upgrade","recipName”:”####”,”senderId”:”######”,”recipId”:”#########”},”user”:”########”,”level":"info","message":"Ran cloud function sendpush for user ########## with:\n Input: {"matchId\”:\”#########\”,\”message":"Test post upgrade","recipName\”:\”#####\”,\”senderId\”:\”######\”,\”recipId\”:\”#######\”}\n Result: "Message was sent successfully"","timestamp":"2019-02-28T14:09:19.771Z"}
{"result":{"response":{"result":"Message was sent successfully"}},"level":"verbose","message":"RESPONSE from [POST] /DateSet/functions/sendpush: {\n "response": {\n "result": "Message was sent successfully"\n }\n}","timestamp":"2019-02-28T14:09:19.772Z"}

{"method":"POST","url":"/dateset/push","headers":{"user-agent":"node-XMLHttpRequest, Parse/js1.9.2 (NodeJS 5.12.0)","accept":"/","content-type":"text/plain","host":"ec2-#############.compute-1.amazonaws.com:5656","content-length":"329","connection":"close"},"body":{"data":{"alert":"Matt: Test 3","badge":"Increment","sound":"default","objectId”:”#######”},”where":{"user":{"__type":"Pointer","className":"_User","objectId”:”#######”}}},”level":"verbose","message":"REQUEST for [POST] /dateset/push: {\n "data": {\n "alert": "Matt: Test 3",\n "badge": "Increment",\n "sound": "default",\n "objectId": \”########\”\n },\n "where": {\n "user": {\n "__type": "Pointer",\n "className": "_User",\n "objectId": \”#########\”\n }\n }\n}","timestamp":"2019-02-28T14:12:00.704Z"}
{"result":{"headers":{"X-Parse-Push-Status-Id":"DnEZQDhi4L"},"response":{"result":true}},"level":"verbose","message":"RESPONSE from [POST] /dateset/push: {\n "headers": {\n "X-Parse-Push-Status-Id": "DnEZQDhi4L"\n },\n "response": {\n "result": true\n }\n}","timestamp":"2019-02-28T14:12:00.711Z"}
{"functionName":"sendpush","params":{"matchId”:”#######”,”message":"Test 3","recipName”:”######”,”senderId”:”#######”,”recipId”:”######”},”user”:”#######”,”level":"info","message":"Ran cloud function sendpush for user ####### with:\n Input: {"matchId\”:\”#########\”,\”message":"Test 3","recipName\”:\”####\”,\”senderId\”:\”########\”,\”recipId\”:\”#########\”}\n Result: "Message was sent successfully"","timestamp":"2019-02-28T14:12:00.718Z"}
{"result":{"response":{"result":"Message was sent successfully"}},"level":"verbose","message":"RESPONSE from [POST] /DateSet/functions/sendpush: {\n "response": {\n "result": "Message was sent successfully"\n }\n}","timestamp":"2019-02-28T14:12:00.719Z"}
{"level":"verbose","message":"sending push to 3 installations","timestamp":"2019-02-28T14:12:00.720Z"}

{"method":"POST","url":"/dateset/push","headers":{"user-agent":"node-XMLHttpRequest, Parse/js1.9.2 (NodeJS 5.12.0)","accept":"/","content-type":"text/plain","host":"ec2-##########.compute-1.amazonaws.com:5656","content-length":"329","connection":"close"},"body":{"data":{"alert":"Matt: Test 4","badge":"Increment","sound":"default","objectId":"8QZWnR0lZb"},"where":{"user":{"__type":"Pointer","className":"_User","objectId”:”########”}}},”level":"verbose","message":"REQUEST for [POST] /dateset/push: {\n "data": {\n "alert": "Matt: Test 4",\n "badge": "Increment",\n "sound": "default",\n "objectId": "8QZWnR0lZb"\n },\n "where": {\n "user": {\n "__type": "Pointer",\n "className": "_User",\n "objectId": \”########\”\n }\n }\n}","timestamp":"2019-02-28T14:20:11.373Z"}
{"result":{"headers":{"X-Parse-Push-Status-Id":"wDHAKqNW3W"},"response":{"result":true}},"level":"verbose","message":"RESPONSE from [POST] /dateset/push: {\n "headers": {\n "X-Parse-Push-Status-Id": "wDHAKqNW3W"\n },\n "response": {\n "result": true\n }\n}","timestamp":"2019-02-28T14:20:11.379Z"}
{"functionName":"sendpush","params":{"matchId”:”#######”,”message":"Test 4","recipName”:”######”,”senderId”:”########”,”recipId”:”#######”},”user”:”########”,”level":"info","message":"Ran cloud function sendpush for user ###### with:\n Input: {"matchId\”:\”#######\”,\”message":"Test 4","recipName\”:\”####\”,\”senderId\”:\”########\”,\”recipId\”:\”#########\”}\n Result: "Message was sent successfully"","timestamp":"2019-02-28T14:20:11.386Z"}
{"result":{"response":{"result":"Message was sent successfully"}},"level":"verbose","message":"RESPONSE from [POST] /DateSet/functions/sendpush: {\n "response": {\n "result": "Message was sent successfully"\n }\n}","timestamp":"2019-02-28T14:20:11.387Z"}
{"level":"verbose","message":"sending push to 3 installations","timestamp":"2019-02-28T14:20:11.390Z"}

[mtrezza]

The push adapter logging seems to be missing here. The phrase parse-server-push-adapter should be contained. See the example here.

Try to set VERBOSE=1 and VERBOSE_PARSE_SERVER_PUSH_ADAPTER=1; that is redundant, but anyway.

[ssgsgugel]

Okay, so when you say set VERBOSE=1. We set logLevel: 1 in "send push" function in our javascript. In addition, we set logLevel: VERBOSE in our index.js in the ParseServer creation. Is this correct? Does this go within the push area?

we run our parse server through the PM command, so are the variables you are talking about environmental defined at the command line?

Let us know, thanks!

[mtrezza]

These are environment variables. See here.

[georgesjamous]

@ssgsgugel

Will enable verbose logging; however, I can tell you our APNS certificate has not expired.

Have you tried sending push notifications from an alternative source, other than parse-server?
I haven't dug into this thread much, but from what I can tell, if you haven't changed anything in your environment, aws instances or code, there is just no reason for the notifications to stop working by themselves.

To properly diagnose this issue, send a notification to a device from a second source, and make sure you receive it. You can use something like pusher.
If it's working, it could be from parse-notification-adapter. Try using another adapter just to make sure it's working from your environment. If that works, then it's definitely from the adapter and you can dive into the code/logs more deeply.

And remember, your aws security groups could be blocking the notification ports, check them out too.

Hope this helps...

[ssgsgugel]

Okay, now that I have found the correct log files and with the environment variables set, here is what I see occurring. Does this usually mean the .p12 file is incorrect/bad, or is this something else like a port connection failure? If it's the .p12, I swear we just created it so that it was current in case that is the problem.

/home/ubuntu/parse-server-example/node_modules/parse-server/lib/ParseServer.js:437
throw err;
^

Error: mac verify failure
at Error (native)
at Object.createSecureContext (_tls_common.js:136:17)
at Object.TLSSocket._init.ssl.onclienthello.ssl.oncertcb.exports.connect (_tls_wrap.js:1007:48)
at apnSocket (/home/ubuntu/parse-server-example/node_modules/apn/lib/socket.js:56:19)
at Connection. (/home/ubuntu/parse-server-example/node_modules/apn/lib/connection.js:199:17)
at _fulfilled (/home/ubuntu/parse-server-example/node_modules/q/q.js:834:54)
at self.promiseDispatch.done (/home/ubuntu/parse-server-example/node_modules/q/q.js:863:30)
at Promise.promise.promiseDispatch (/home/ubuntu/parse-server-example/node_modules/q/q.js:796:13)
at /home/ubuntu/parse-server-example/node_modules/q/q.js:604:44
at runSingle (/home/ubuntu/parse-server-example/node_modules/q/q.js:137:13)
ERR! parse-server-push-adapter APNS cannot find vaild connection for 9d2dced6bc5f3079928edc3317a36ccc7125a396997b62e91421fe84e8abf5e8
ERR! parse-server-push-adapter APNS cannot find vaild connection for dd5f84e01838117ecaa9d28d25d184f10366f3adf515333bdbf097eaa66514f7
Error: mac verify failure
at Error (native)
at Object.createSecureContext (_tls_common.js:136:17)
at Object.TLSSocket._init.ssl.onclienthello.ssl.oncertcb.exports.connect (_tls_wrap.js:1007:48)
at apnSocket (/home/ubuntu/parse-server-example/node_modules/apn/lib/socket.js:56:19)
at Connection. (/home/ubuntu/parse-server-example/node_modules/apn/lib/connection.js:199:17)
at _fulfilled (/home/ubuntu/parse-server-example/node_modules/q/q.js:834:54)
at self.promiseDispatch.done (/home/ubuntu/parse-server-example/node_modules/q/q.js:863:30)
at Promise.promise.promiseDispatch (/home/ubuntu/parse-server-example/node_modules/q/q.js:796:13)
at /home/ubuntu/parse-server-example/node_modules/q/q.js:604:44
at runSingle (/home/ubuntu/parse-server-example/node_modules/q/q.js:137:13)
/home/ubuntu/parse-server-example/node_modules/parse-server/lib/ParseServer.js:437
throw err;
^

Error: mac verify failure
at Error (native)
at Object.createSecureContext (_tls_common.js:136:17)
90509,1 99%

[georgesjamous]

APNS cannot find vaild connection for

I believe it has something to do with the connection, so either the .p12 file has expired, the wrong environment is being selected (production/development) or a port problem.
Read my previous comment, it will help you narrow down the source of the problem, it also might be a bug in the adapter.

Also, your issue is better suited for the push-notification adapter repo. I recommend you re-open it there.

There are issues regarding
#21
#1665

[ssgsgugel]

Moved per suggestion to the push-adapter repo.