From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1758915AbcJ1Nxt convert rfc822-to-8bit (ORCPT ); Fri, 28 Oct 2016 09:53:49 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:37929 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752143AbcJ1Nxp (ORCPT ); Fri, 28 Oct 2016 09:53:45 -0400 Subject: Re: [PATCH v2 02/16] scsi: don't use fc_bsg_job::request and fc_bsg_job::reply directly To: Hannes Reinecke , Johannes Thumshirn , Ulrich Weigand , Andreas Krebbel References: <2ea07f3f-88eb-b795-fa37-a223bf80e581@linux.vnet.ibm.com> <20161013162405.aoxy3bdkc4bqtwsk@linux-x5ow.site> <4b411836-e76f-b67a-3d49-ad3d51b8f216@linux.vnet.ibm.com> Cc: "Martin K . Petersen" , Christoph Hellwig , Linux Kernel Mailinglist , Linux SCSI Mailinglist , Martin Schwidefsky , Heiko Carstens , Anil Gurumurthy , Sudarsana Kalluru , "James E.J. Bottomley" , Tyrel Datwyler , Benjamin Herrenschmidt , Paul Mackerras , Michael Ellerman , Johannes Thumshirn , James Smart , Dick Kennedy , "supporter:QLOGIC QLA2XXX FC-SCSI DRIVER" , "open list:S390 ZFCP DRIVER" , "open list:LINUX FOR POWERPC (32-BIT AND 64-BIT)" , "open list:FCOE SUBSYSTEM (libfc, libfcoe, fcoe)" , Richard Biener From: Steffen Maier Date: Fri, 28 Oct 2016 15:53:33 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 8BIT X-TM-AS-MML: disable X-Content-Scanned: Fidelis XPS MAILER x-cbid: 16102813-0036-0000-0000-0000024E8CDC X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 16102813-0037-0000-0000-0000130AC89D Message-Id: <132d8dc4-e8d0-6eba-9ae2-4a7e2c9a589b@linux.vnet.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2016-10-28_05:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 suspectscore=0 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1609300000 definitions=main-1610280235 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 10/28/2016 01:31 PM, Hannes Reinecke wrote: > On 10/28/2016 11:53 AM, Steffen Maier wrote: >> On 10/13/2016 06:24 PM, Johannes Thumshirn wrote: >>> On Thu, Oct 13, 2016 at 05:15:25PM +0200, Steffen Maier wrote: >>>> I'm puzzled. >>>> >>>> $ git bisect start fc_bsg master >> >>>>> 3087864ce3d7282f59021245d8a5f83ef1caef18 is the first bad commit >>>>> commit 3087864ce3d7282f59021245d8a5f83ef1caef18 >>>>> Author: Johannes Thumshirn >>>>> Date: Wed Oct 12 15:06:28 2016 +0200 >>>>> >>>>> scsi: don't use fc_bsg_job::request and fc_bsg_job::reply directly >>>>> >>>>> Don't use fc_bsg_job::request and fc_bsg_job::reply directly, >>>>> but use >>>>> helper variables bsg_request and bsg_reply. This will be >>>>> helpfull when >>>>> transitioning to bsg-lib. >>>>> >>>>> Signed-off-by: Johannes Thumshirn >>>>> >>>>> :040000 040000 140c4b6829d5cfaec4079716e0795f63f8bc3bd2 >>>>> 0d9fe225615679550be91fbd9f84c09ab1e280fc M drivers >>>> >>>> From there (on the reverse bisect path) I get the following Oops, >>>> except for the full patch set having another stack trace as in my >>>> previous >>>> mail (dying in zfcp code). >>> >>> [...] >>> >>>>> @@ -3937,6 +3944,7 @@ fc_bsg_request_handler(struct request_queue >>>>> *q, struct Scsi_Host *shost, >>>>> struct request *req; >>>>> struct fc_bsg_job *job; >>>>> enum fc_dispatch_result ret; >>>>> + struct fc_bsg_reply *bsg_reply; >>>>> >>>>> if (!get_device(dev)) >>>>> return; >>>>> @@ -3973,8 +3981,9 @@ fc_bsg_request_handler(struct request_queue >>>>> *q, struct Scsi_Host *shost, >>>>> /* check if we have the msgcode value at least */ >>>>> if (job->request_len < sizeof(uint32_t)) { >>>>> BUG_ON(job->reply_len < sizeof(uint32_t)); >>>>> - job->reply->reply_payload_rcv_len = 0; >>>>> - job->reply->result = -ENOMSG; >>>>> + bsg_reply = job->reply; >>>>> + bsg_reply->reply_payload_rcv_len = 0; >>>>> + bsg_reply->result = -ENOMSG; >> >> Compiler optimization re-ordered above two lines and the first pointer >> derefence is bsg_reply->result [field offset 0] where bsg_reply is NULL. >> The assignment tries to write to memory at address NULL causing the >> kernel page fault. >> > I spoke to our compiler people, and they strongly believed this not to > be the case. Or, put it the other way round, if such a thing would > happen it would be a compiler issue. > > Have you checked the compiler output? I just mentioned the compiler optimization to explain why the assembler code visible in the panic dies at bsg_reply->result = -ENOMSG and not at bsg_reply->reply_payload_rcv_len = 0. I don't think it makes a difference regarding the issue, which remains a NULL pointer dereference with bsg_reply either way, which I doubt is caused by compiler output. But then again, see further down below. > [ 46.942560] Krnl PSW : 0704e00180000000 00000000007c91ec[ 46.942574] (fc_bsg_request_handler+0x404/0x4b0) > [ 46.942579] > [ 46.942583] R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:2 PM:000: > [ 46.942598] RI:0 EA:3 > [ 46.942601] > [ 46.942601] Krnl GPRS: 0000000000000000 00000000ffffffcb 0000000000000000 0000000080000001 > [ 46.942603] 00000000007c8fe8 0000000064398c68 0000000069f967e8 000000006a3d8008 > [ 46.942605] 000000006a5e02c8 00000000698b5490 0000000000000000 0000000000000000 %r11 is NULL > [ 46.942607] 000000006a9ef5f8 0000000000a36840 00000000007c8fe8 000000005d2efa00 > [ 46.942619] Krnl Code: 00000000007c91de: e55dc08c0003 clfhsi 140(%r12),3[ 46.942622] > [ 46.942622] 00000000007c91e4: a7240004 brc 2,7c91ec > #00000000007c91e8: a7f40001 brc 15,7c91ea[ 46.942629] > [ 46.942629] >00000000007c91ec: 5010b000 st %r1,0(%r11) > 00000000007c91f0: e54cb0040000 mvhi 4(%r11),0[ 46.942635] > [ 46.942635] 00000000007c91f6: e54cc08c0004 mvhi 140(%r12),4 > 00000000007c91fc: b904002c lgr %r2,%r12[ 46.942643] > [ 46.942643] 00000000007c9200: c0e5ffffe2c0 brasl %r14,7c5780 > [ 46.942646] > [ 46.942647] Call Trace: > [ 46.942650] ([<00000000007c8fe8>] fc_bsg_request_handler+0x200/0x4b0) > [ 46.942656] ([<00000000006b8e0a>] __blk_run_queue+0x52/0x68) > [ 46.942661] ([<00000000006c549a>] blk_execute_rq_nowait+0xf2/0x110) > [ 46.942664] ([<00000000006c557a>] blk_execute_rq+0xa2/0x110) > [ 46.942668] ([<00000000006de0ee>] bsg_ioctl+0x1f6/0x268) > [ 46.942675] ([<000000000036ca20>] do_vfs_ioctl+0x680/0x6d8) > [ 46.942677] ([<000000000036caf4>] SyS_ioctl+0x7c/0xb0) > [ 46.942685] ([<00000000009a541e>] system_call+0xd6/0x270) > [ 46.942687] INFO: lockdep is turned off. > [ 46.942688] Last Breaking-Event-Address: > [ 46.942692] [<00000000007c91e4>] fc_bsg_request_handler+0x3fc/0x4b0 > [ 46.942696] [ 46.942698] Kernel panic - not syncing: Fatal exception: panic_on_oops all the following was written from bottom to top: > crash> dis -l fc_bsg_request_handler > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3943 static void fc_bsg_request_handler(struct request_queue *q, struct Scsi_Host *shost, struct fc_rport *rport, struct device *dev) { > 0x7c8de8 : brcl 0,0x7c8de8 > 0x7c8dee : stmg %r6,%r15,72(%r15) > 0x7c8df4 : larl %r13,0xa36840 > 0x7c8dfa : tmll %r15,16256 > 0x7c8dfe : lgr %r14,%r15 > 0x7c8e02 : je 0x7c8e04 > 0x7c8e06 : lay %r15,-112(%r15) > 0x7c8e0c : stg %r14,152(%r15) > 0x7c8e12 : lgr %r9,%r2 > 0x7c8e16 : stg %r5,176(%r15) > 0x7c8e1c : lgr %r2,%r5 > 0x7c8e20 : lgr %r6,%r3 > 0x7c8e24 : lgr %r10,%r4 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3949 > 0x7c8e28 : brasl %r14,0x787968 > 0x7c8e2e : cgij %r2,0,8,0x7c9288 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3751 there is some confusing inlining of part of fc_req_to_bsgjob > 0x7c8e34 : la %r1,960(%r6) > 0x7c8e38 : stg %r1,168(%r15) > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3749 > 0x7c8e3e : la %r1,96(%r10) > 0x7c8e42 : stg %r1,160(%r15) > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3953 > 0x7c8e48 : cgij %r10,0,8,0x7c9270 > 0x7c8e4e : clc 4(4,%r13),40(%r10) > 0x7c8e54 : jne 0x7c9258 > 0x7c8e58 : tm 72(%r10),4 > 0x7c8e5c : jne 0x7c9258 > 0x7c8e60 : j 0x7c920a > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3961 > 0x7c8e64 : clc 0(4,%r13),40(%r10) > 0x7c8e6a : je 0x7c8e9e > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3962 fc_bsg_request_handler() req->errors = -ENXIO; > 0x7c8e6e : mvhi 260(%r12),-6 crash> struct -od request.errors struct request { [260] int errors; } ******************************************************************** BUT this seems the first time %r12 is used in fc_bsg_request_handler(), especially I seem to miss %r12 being initalized with anything. But then again I'm not at all well versed in disassembly. Maybe fc_bsg_request_handler() is itself in turn inlined and I would need to start disassembling even earlier to get to %r12 init? s390x ELF ABI says %r12: usage: Local variable, commonly used as GOT pointer; call effect: saved. Even if it wasn't initialized and remained NULL below why did it not already page fault at above instruction? Silly me, we did not execute this instruction as it's "if" conditional. This makes me wonder even more where the content of %r12 comes from. Ulli, Andreas, could you please shed some light on this? ******************************************************************** > /home/maier/kernel/linux-vanilla/./include/linux/spinlock.h: 357 > 0x7c8e74 : lg %r2,2600(%r9) > 0x7c8e7a : brasl %r14,0x9a46d0 <_raw_spin_unlock_irq> > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3964 > 0x7c8e80 : lgr %r2,%r12 > 0x7c8e84 : lghi %r3,-6 > 0x7c8e88 : brasl %r14,0x6be2f0 > /home/maier/kernel/linux-vanilla/./include/linux/spinlock.h: 332 > 0x7c8e8e : lg %r2,2600(%r9) > 0x7c8e94 : brasl %r14,0x9a4280 <_raw_spin_lock_irq> > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3966 > 0x7c8e9a : j 0x7c8e48 > /home/maier/kernel/linux-vanilla/./include/linux/spinlock.h: 357 > 0x7c8e9e : lg %r2,2600(%r9) > 0x7c8ea4 : brasl %r14,0x9a46d0 <_raw_spin_unlock_irq> > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3709 > 0x7c8eaa : ltg %r1,248(%r12) > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3702 > 0x7c8eb0 : lg %r7,512(%r6) > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3703 > 0x7c8eb6 : lg %r8,360(%r12) > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3709 > 0x7c8ebc : je 0x7c8ec4 > 0x7c8ec0 : j 0x7c8ec2 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3711 > 0x7c8ec4 : lg %r1,568(%r7) > 0x7c8eca : llgf %r1,216(%r1) > /home/maier/kernel/linux-vanilla/./include/linux/slab.h: 495 > 0x7c8ed0 : lgfi %r3,37781696 > 0x7c8ed6 : la %r2,184(%r1) > 0x7c8eda : brasl %r14,0x325e38 <__kmalloc> > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3713 > 0x7c8ee0 : lgr %r11,%r2 > 0x7c8ee4 : cgij %r2,0,8,0x7c9234 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3725 fc_req_to_bsgjob() req->special = job; > 0x7c8eea : stg %r2,248(%r12) > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3726 > 0x7c8ef0 : stg %r6,0(%r2) > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3727 > 0x7c8ef6 : stg %r10,8(%r2) > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3728 fc_req_to_bsgjob() job->req = req; > 0x7c8efc : stg %r12,24(%r2) > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3729 > 0x7c8f02 : lg %r1,568(%r7) > 0x7c8f08 : lt %r1,216(%r1) > 0x7c8f0e : je 0x7c8f1c > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3730 > 0x7c8f12 : la %r1,184(%r2) > 0x7c8f16 : stg %r1,176(%r2) > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3731 > 0x7c8f1c : larl %r4,0x2054808 > 0x7c8f22 : larl %r3,0xbddbd8 > 0x7c8f28 : la %r2,32(%r11) > 0x7c8f2c : brasl %r14,0x1b7ac8 <__raw_spin_lock_init> > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3733 > 0x7c8f32 : llh %r1,288(%r12) > 0x7c8f38 : st %r1,136(%r11) > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3735 > 0x7c8f3c : mvhi 140(%r11),96 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3737 > 0x7c8f42 : ltg %r1,104(%r12) > 0x7c8f48 : jne 0x7c8f56 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3742 > 0x7c8f4c : cgij %r8,0,6,0x7c8f84 > 0x7c8f52 : j 0x7c8f6e > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3738 > 0x7c8f56 : lgr %r3,%r12 > 0x7c8f5a : la %r2,144(%r11) > 0x7c8f5e : brasl %r14,0x7c56c8 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3739 > 0x7c8f64 : cij %r2,0,8,0x7c8f4c > 0x7c8f6a : j 0x7c900e > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3747 > 0x7c8f6e : larl %r1,0x7c5780 > 0x7c8f74 : stg %r1,112(%r11) > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3748 > 0x7c8f7a : cgij %r10,0,6,0x7c8fa6 > 0x7c8f80 : j 0x7c8fd2 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3742 > 0x7c8f84 : ltg %r1,104(%r8) > 0x7c8f8a : je 0x7c8f6e > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3743 > 0x7c8f8e : lgr %r3,%r8 > 0x7c8f92 : la %r2,160(%r11) > 0x7c8f96 : brasl %r14,0x7c56c8 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3744 > 0x7c8f9c : cij %r2,0,8,0x7c8f6e > 0x7c8fa2 : j 0x7c9002 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3749 > 0x7c8fa6 : lg %r2,160(%r15) > 0x7c8fac : stg %r2,16(%r11) > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3752 > 0x7c8fb2 : brasl %r14,0x787968 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3754 > 0x7c8fb8 : mvhi 108(%r11),1 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3979 fc_bsg_request_handler() job = req->special; > 0x7c8fbe : lg %r12,248(%r12) crash> struct -od request.special struct request { [248] void *special; } ******************************************************************** so above %r12 did contain req, below it contains job. since we could deref req further up it must have been non-NULL and pointing to a mapped page, but req->special is NULL here? well, req could even have been NULL and we read from address 248 in low core here which does not trigger a page fault (only on write to low core). crash> x/g 248 0xf8 <_text+248>: 0x0 ******************************************************************** > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3982 > 0x7c8fc4 : l %r1,136(%r12) > 0x7c8fc8 : clij %r1,3,12,0x7c901c > 0x7c8fce : j 0x7c905c > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3751 > 0x7c8fd2 : lg %r1,168(%r15) > 0x7c8fd8 : stg %r1,16(%r11) > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3752 > 0x7c8fde : lgr %r2,%r1 > 0x7c8fe2 : brasl %r14,0x787968 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3754 > 0x7c8fe8 : mvhi 108(%r11),1 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3979 > 0x7c8fee : lg %r12,248(%r12) > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3982 > 0x7c8ff4 : l %r1,136(%r12) > 0x7c8ff8 : clij %r1,3,12,0x7c901c > 0x7c8ffe : j 0x7c90f4 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3760 > 0x7c9002 : lg %r2,152(%r11) > 0x7c9008 : brasl %r14,0x328ff0 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3762 > 0x7c900e : lgr %r2,%r11 > 0x7c9012 : brasl %r14,0x328ff0 > 0x7c9018 : j 0x7c9234 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3983 > 0x7c901c : clfhsi 140(%r12),3 > 0x7c9022 : jh 0x7c902a > 0x7c9026 : j 0x7c9028 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3984 > 0x7c902a : lg %r1,128(%r12) > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3985 > 0x7c9030 : mvhi 4(%r1),0 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3986 > 0x7c9036 : mvhi 0(%r1),-42 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3988 > 0x7c903c : lgr %r2,%r12 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3987 > 0x7c9040 : mvhi 140(%r12),4 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3988 > 0x7c9046 : brasl %r14,0x7c5780 > /home/maier/kernel/linux-vanilla/./include/linux/spinlock.h: 332 > 0x7c904c : lg %r2,2600(%r9) > 0x7c9052 : brasl %r14,0x9a4280 <_raw_spin_lock_irq> > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3990 > 0x7c9058 : j 0x7c8e48 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3892 > 0x7c905c : lg %r2,120(%r12) > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3887 > 0x7c9062 : lg %r11,128(%r12) > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3892 > 0x7c9068 : l %r2,0(%r2) > 0x7c906c : iilf %r3,1073741825 > 0x7c9072 : crj %r2,%r3,8,0x7c9088 > 0x7c9078 : iilf %r3,1073741826 > 0x7c907e : crj %r2,%r3,8,0x7c9090 > 0x7c9084 : j 0x7c90d2 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3894 > 0x7c9088 : lhi %r2,5 > 0x7c908c : j 0x7c9094 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3898 > 0x7c9090 : lhi %r2,16 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3901 > 0x7c9094 : lt %r3,144(%r12) > 0x7c909a : je 0x7c90da > 0x7c909e : lt %r3,160(%r12) > 0x7c90a4 : je 0x7c90da > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3913 > 0x7c90a8 : clrj %r2,%r1,2,0x7c90e2 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3918 > 0x7c90ae : lg %r1,512(%r6) > 0x7c90b4 : lg %r1,568(%r1) > 0x7c90ba : lg %r1,192(%r1) > 0x7c90c0 : lgr %r2,%r12 > 0x7c90c4 : basr %r14,%r1 > 0x7c90c6 : lr %r1,%r2 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3919 > 0x7c90c8 : cij %r2,0,6,0x7c90e6 > 0x7c90ce : j 0x7c9248 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3908 > 0x7c90d2 : lhi %r1,-53 > 0x7c90d6 : j 0x7c90e6 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3903 > 0x7c90da : lhi %r1,-22 > 0x7c90de : j 0x7c90e6 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3914 > 0x7c90e2 : lhi %r1,-42 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3924 > 0x7c90e6 : clfhsi 140(%r12),3 > 0x7c90ec : jh 0x7c91ec > 0x7c90f0 : j 0x7c90f2 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3785 fc_bsg_host_dispatch() struct fc_bsg_request *bsg_request = job->request; > 0x7c90f4 : lg %r3,120(%r12) > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3786 fc_bsg_host_dispatch() struct fc_bsg_reply *bsg_reply = job->reply; > 0x7c90fa : lg %r11,128(%r12) load content of address in %r12 with displacement 128 into %r11. so presumably job->reply is NULL. due to funny inlining incl. fc_bsg_host_dispatch(), it's tricky to backtrack where job in %r12 came from and what happened to it on the way. %r11 is not clobbered until used below where the page fault happens. displacement is consistent: crash> struct -od fc_bsg_job struct fc_bsg_job { [0] struct Scsi_Host *shost; [8] struct fc_rport *rport; [16] struct device *dev; [24] struct request *req; [32] spinlock_t job_lock; [104] unsigned int state_flags; [108] unsigned int ref_cnt; [112] void (*job_done)(struct fc_bsg_job *); [120] struct fc_bsg_request *request; [128] struct fc_bsg_reply *reply; [136] unsigned int request_len; [140] unsigned int reply_len; [144] struct bsg_buffer request_payload; [160] struct bsg_buffer reply_payload; [176] void *dd_data; } SIZE: 184 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3791 > 0x7c9100 : l %r2,0(%r3) > 0x7c9104 : clfi %r2,2147483651 > 0x7c910a : je 0x7c913e > 0x7c910e : jh 0x7c9122 > 0x7c9112 : iilf %r3,2147483649 > 0x7c9118 : clrj %r2,%r3,10,0x7c9194 > 0x7c911e : j 0x7c91c2 > 0x7c9122 : iilf %r4,2147483652 > 0x7c9128 : crj %r2,%r4,8,0x7c9156 > 0x7c912e : iilf %r4,2147483903 > 0x7c9134 : crj %r2,%r4,8,0x7c9172 > 0x7c913a : j 0x7c91c2 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3803 > 0x7c913e : lt %r2,144(%r12) > 0x7c9144 : je 0x7c91ca > 0x7c9148 : lt %r2,160(%r12) > 0x7c914e : je 0x7c91ca > 0x7c9152 : j 0x7c9194 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3813 > 0x7c9156 : lt %r2,144(%r12) > 0x7c915c : je 0x7c91ca > 0x7c9160 : lt %r2,160(%r12) > 0x7c9166 : je 0x7c91ca > 0x7c916a : lhi %r2,20 > 0x7c916e : j 0x7c9198 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3822 > 0x7c9172 : lg %r2,504(%r6) > 0x7c9178 : ltg %r2,304(%r2) > 0x7c917e : je 0x7c91d2 > 0x7c9182 : cg %r2,4(%r3) > 0x7c9188 : jne 0x7c91d2 > 0x7c918c : lhi %r2,12 > 0x7c9190 : j 0x7c9198 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3791 > 0x7c9194 : lhi %r2,8 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3836 > 0x7c9198 : clrj %r2,%r1,2,0x7c91da > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3841 > 0x7c919e : lg %r1,512(%r6) > 0x7c91a4 : lg %r1,568(%r1) > 0x7c91aa : lg %r1,192(%r1) > 0x7c91b0 : lgr %r2,%r12 > 0x7c91b4 : basr %r14,%r1 > 0x7c91b6 : lr %r1,%r2 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3842 > 0x7c91b8 : cij %r2,0,6,0x7c91de > 0x7c91be : j 0x7c9248 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3831 > 0x7c91c2 : lhi %r1,-53 > 0x7c91c6 : j 0x7c91de > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3805 > 0x7c91ca : lhi %r1,-22 > 0x7c91ce : j 0x7c91de > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3825 > 0x7c91d2 : lhi %r1,-3 > 0x7c91d6 : j 0x7c91de > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3837 > 0x7c91da : lhi %r1,-42 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3847 fc_bsg_host_dispatch() fail_host_msg: /* return the errno failure code as the only status */ BUG_ON(job->reply_len < sizeof(uint32_t)); > 0x7c91de : clfhsi 140(%r12),3 > 0x7c91e4 : jh 0x7c91ec > 0x7c91e8 : j 0x7c91ea > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3849 bsg_reply->result = ret; > 0x7c91ec : st %r1,0(%r11) that store causes the kernel page fault because %r11 is NULL and with displacement 0 it still is NULL. > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3848 bsg_reply->reply_payload_rcv_len = 0; > 0x7c91f0 : mvhi 4(%r11),0 if we would have gotten this far: 16-bit signed immediate 0 is extended to 4-bytes and stored to where %r11 with displacement 4 points to. displacements nicely match structure fields: crash> struct -od fc_bsg_reply struct fc_bsg_reply { [0] uint32_t result; [4] uint32_t reply_payload_rcv_len; union { struct fc_bsg_host_vendor_reply vendor_reply; struct fc_bsg_ctels_reply ctels_reply; [8] } reply_data; } SIZE: 16 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3850 job->reply_len = sizeof(uint32_t); > 0x7c91f6 : mvhi 140(%r12),4 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3851 > 0x7c91fc : lgr %r2,%r12 > 0x7c9200 : brasl %r14,0x7c5780 source code is based on > $ git log --graph --oneline > * 271c1723d9c8 scsi: don't use fc_bsg_job::request and fc_bsg_job::reply directly > * a3c95a6c69e4 scsi: Get rid of struct fc_bsg_buffer > * 1573d2caf713 Merge branch 'parisc-4.9-2' of git://git.kernel.org/pub/scm/linux/kernel/git/deller/parisc-linux -- Mit freundlichen Grüßen / Kind regards Steffen Maier Linux on z Systems Development IBM Deutschland Research & Development GmbH Vorsitzende des Aufsichtsrats: Martina Koederitz Geschaeftsfuehrung: Dirk Wittkopp Sitz der Gesellschaft: Boeblingen Registergericht: Amtsgericht Stuttgart, HRB 243294 From mboxrd@z Thu Jan 1 00:00:00 1970 From: Steffen Maier Subject: Re: [PATCH v2 02/16] scsi: don't use fc_bsg_job::request and fc_bsg_job::reply directly Date: Fri, 28 Oct 2016 15:53:33 +0200 Message-ID: <132d8dc4-e8d0-6eba-9ae2-4a7e2c9a589b@linux.vnet.ibm.com> References: <2ea07f3f-88eb-b795-fa37-a223bf80e581@linux.vnet.ibm.com> <20161013162405.aoxy3bdkc4bqtwsk@linux-x5ow.site> <4b411836-e76f-b67a-3d49-ad3d51b8f216@linux.vnet.ibm.com> Mime-Version: 1.0 Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 8BIT Return-path: In-Reply-To: Sender: linux-kernel-owner@vger.kernel.org To: Hannes Reinecke , Johannes Thumshirn , Ulrich Weigand , Andreas Krebbel Cc: "Martin K . Petersen" , Christoph Hellwig , Linux Kernel Mailinglist , Linux SCSI Mailinglist , Martin Schwidefsky , Heiko Carstens , Anil Gurumurthy , Sudarsana Kalluru , "James E.J. Bottomley" , Tyrel Datwyler , Benjamin Herrenschmidt , Paul Mackerras , Michael Ellerman , Johannes Thumshirn , James Smart , Dick Kennedy , supporter:QLOGIC QLA2XXX FC-SCSI DRIVER List-Id: linux-scsi@vger.kernel.org On 10/28/2016 01:31 PM, Hannes Reinecke wrote: > On 10/28/2016 11:53 AM, Steffen Maier wrote: >> On 10/13/2016 06:24 PM, Johannes Thumshirn wrote: >>> On Thu, Oct 13, 2016 at 05:15:25PM +0200, Steffen Maier wrote: >>>> I'm puzzled. >>>> >>>> $ git bisect start fc_bsg master >> >>>>> 3087864ce3d7282f59021245d8a5f83ef1caef18 is the first bad commit >>>>> commit 3087864ce3d7282f59021245d8a5f83ef1caef18 >>>>> Author: Johannes Thumshirn >>>>> Date: Wed Oct 12 15:06:28 2016 +0200 >>>>> >>>>> scsi: don't use fc_bsg_job::request and fc_bsg_job::reply directly >>>>> >>>>> Don't use fc_bsg_job::request and fc_bsg_job::reply directly, >>>>> but use >>>>> helper variables bsg_request and bsg_reply. This will be >>>>> helpfull when >>>>> transitioning to bsg-lib. >>>>> >>>>> Signed-off-by: Johannes Thumshirn >>>>> >>>>> :040000 040000 140c4b6829d5cfaec4079716e0795f63f8bc3bd2 >>>>> 0d9fe225615679550be91fbd9f84c09ab1e280fc M drivers >>>> >>>> From there (on the reverse bisect path) I get the following Oops, >>>> except for the full patch set having another stack trace as in my >>>> previous >>>> mail (dying in zfcp code). >>> >>> [...] >>> >>>>> @@ -3937,6 +3944,7 @@ fc_bsg_request_handler(struct request_queue >>>>> *q, struct Scsi_Host *shost, >>>>> struct request *req; >>>>> struct fc_bsg_job *job; >>>>> enum fc_dispatch_result ret; >>>>> + struct fc_bsg_reply *bsg_reply; >>>>> >>>>> if (!get_device(dev)) >>>>> return; >>>>> @@ -3973,8 +3981,9 @@ fc_bsg_request_handler(struct request_queue >>>>> *q, struct Scsi_Host *shost, >>>>> /* check if we have the msgcode value at least */ >>>>> if (job->request_len < sizeof(uint32_t)) { >>>>> BUG_ON(job->reply_len < sizeof(uint32_t)); >>>>> - job->reply->reply_payload_rcv_len = 0; >>>>> - job->reply->result = -ENOMSG; >>>>> + bsg_reply = job->reply; >>>>> + bsg_reply->reply_payload_rcv_len = 0; >>>>> + bsg_reply->result = -ENOMSG; >> >> Compiler optimization re-ordered above two lines and the first pointer >> derefence is bsg_reply->result [field offset 0] where bsg_reply is NULL. >> The assignment tries to write to memory at address NULL causing the >> kernel page fault. >> > I spoke to our compiler people, and they strongly believed this not to > be the case. Or, put it the other way round, if such a thing would > happen it would be a compiler issue. > > Have you checked the compiler output? I just mentioned the compiler optimization to explain why the assembler code visible in the panic dies at bsg_reply->result = -ENOMSG and not at bsg_reply->reply_payload_rcv_len = 0. I don't think it makes a difference regarding the issue, which remains a NULL pointer dereference with bsg_reply either way, which I doubt is caused by compiler output. But then again, see further down below. > [ 46.942560] Krnl PSW : 0704e00180000000 00000000007c91ec[ 46.942574] (fc_bsg_request_handler+0x404/0x4b0) > [ 46.942579] > [ 46.942583] R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:2 PM:000: > [ 46.942598] RI:0 EA:3 > [ 46.942601] > [ 46.942601] Krnl GPRS: 0000000000000000 00000000ffffffcb 0000000000000000 0000000080000001 > [ 46.942603] 00000000007c8fe8 0000000064398c68 0000000069f967e8 000000006a3d8008 > [ 46.942605] 000000006a5e02c8 00000000698b5490 0000000000000000 0000000000000000 %r11 is NULL > [ 46.942607] 000000006a9ef5f8 0000000000a36840 00000000007c8fe8 000000005d2efa00 > [ 46.942619] Krnl Code: 00000000007c91de: e55dc08c0003 clfhsi 140(%r12),3[ 46.942622] > [ 46.942622] 00000000007c91e4: a7240004 brc 2,7c91ec > #00000000007c91e8: a7f40001 brc 15,7c91ea[ 46.942629] > [ 46.942629] >00000000007c91ec: 5010b000 st %r1,0(%r11) > 00000000007c91f0: e54cb0040000 mvhi 4(%r11),0[ 46.942635] > [ 46.942635] 00000000007c91f6: e54cc08c0004 mvhi 140(%r12),4 > 00000000007c91fc: b904002c lgr %r2,%r12[ 46.942643] > [ 46.942643] 00000000007c9200: c0e5ffffe2c0 brasl %r14,7c5780 > [ 46.942646] > [ 46.942647] Call Trace: > [ 46.942650] ([<00000000007c8fe8>] fc_bsg_request_handler+0x200/0x4b0) > [ 46.942656] ([<00000000006b8e0a>] __blk_run_queue+0x52/0x68) > [ 46.942661] ([<00000000006c549a>] blk_execute_rq_nowait+0xf2/0x110) > [ 46.942664] ([<00000000006c557a>] blk_execute_rq+0xa2/0x110) > [ 46.942668] ([<00000000006de0ee>] bsg_ioctl+0x1f6/0x268) > [ 46.942675] ([<000000000036ca20>] do_vfs_ioctl+0x680/0x6d8) > [ 46.942677] ([<000000000036caf4>] SyS_ioctl+0x7c/0xb0) > [ 46.942685] ([<00000000009a541e>] system_call+0xd6/0x270) > [ 46.942687] INFO: lockdep is turned off. > [ 46.942688] Last Breaking-Event-Address: > [ 46.942692] [<00000000007c91e4>] fc_bsg_request_handler+0x3fc/0x4b0 > [ 46.942696] [ 46.942698] Kernel panic - not syncing: Fatal exception: panic_on_oops all the following was written from bottom to top: > crash> dis -l fc_bsg_request_handler > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3943 static void fc_bsg_request_handler(struct request_queue *q, struct Scsi_Host *shost, struct fc_rport *rport, struct device *dev) { > 0x7c8de8 : brcl 0,0x7c8de8 > 0x7c8dee : stmg %r6,%r15,72(%r15) > 0x7c8df4 : larl %r13,0xa36840 > 0x7c8dfa : tmll %r15,16256 > 0x7c8dfe : lgr %r14,%r15 > 0x7c8e02 : je 0x7c8e04 > 0x7c8e06 : lay %r15,-112(%r15) > 0x7c8e0c : stg %r14,152(%r15) > 0x7c8e12 : lgr %r9,%r2 > 0x7c8e16 : stg %r5,176(%r15) > 0x7c8e1c : lgr %r2,%r5 > 0x7c8e20 : lgr %r6,%r3 > 0x7c8e24 : lgr %r10,%r4 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3949 > 0x7c8e28 : brasl %r14,0x787968 > 0x7c8e2e : cgij %r2,0,8,0x7c9288 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3751 there is some confusing inlining of part of fc_req_to_bsgjob > 0x7c8e34 : la %r1,960(%r6) > 0x7c8e38 : stg %r1,168(%r15) > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3749 > 0x7c8e3e : la %r1,96(%r10) > 0x7c8e42 : stg %r1,160(%r15) > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3953 > 0x7c8e48 : cgij %r10,0,8,0x7c9270 > 0x7c8e4e : clc 4(4,%r13),40(%r10) > 0x7c8e54 : jne 0x7c9258 > 0x7c8e58 : tm 72(%r10),4 > 0x7c8e5c : jne 0x7c9258 > 0x7c8e60 : j 0x7c920a > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3961 > 0x7c8e64 : clc 0(4,%r13),40(%r10) > 0x7c8e6a : je 0x7c8e9e > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3962 fc_bsg_request_handler() req->errors = -ENXIO; > 0x7c8e6e : mvhi 260(%r12),-6 crash> struct -od request.errors struct request { [260] int errors; } ******************************************************************** BUT this seems the first time %r12 is used in fc_bsg_request_handler(), especially I seem to miss %r12 being initalized with anything. But then again I'm not at all well versed in disassembly. Maybe fc_bsg_request_handler() is itself in turn inlined and I would need to start disassembling even earlier to get to %r12 init? s390x ELF ABI says %r12: usage: Local variable, commonly used as GOT pointer; call effect: saved. Even if it wasn't initialized and remained NULL below why did it not already page fault at above instruction? Silly me, we did not execute this instruction as it's "if" conditional. This makes me wonder even more where the content of %r12 comes from. Ulli, Andreas, could you please shed some light on this? ******************************************************************** > /home/maier/kernel/linux-vanilla/./include/linux/spinlock.h: 357 > 0x7c8e74 : lg %r2,2600(%r9) > 0x7c8e7a : brasl %r14,0x9a46d0 <_raw_spin_unlock_irq> > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3964 > 0x7c8e80 : lgr %r2,%r12 > 0x7c8e84 : lghi %r3,-6 > 0x7c8e88 : brasl %r14,0x6be2f0 > /home/maier/kernel/linux-vanilla/./include/linux/spinlock.h: 332 > 0x7c8e8e : lg %r2,2600(%r9) > 0x7c8e94 : brasl %r14,0x9a4280 <_raw_spin_lock_irq> > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3966 > 0x7c8e9a : j 0x7c8e48 > /home/maier/kernel/linux-vanilla/./include/linux/spinlock.h: 357 > 0x7c8e9e : lg %r2,2600(%r9) > 0x7c8ea4 : brasl %r14,0x9a46d0 <_raw_spin_unlock_irq> > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3709 > 0x7c8eaa : ltg %r1,248(%r12) > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3702 > 0x7c8eb0 : lg %r7,512(%r6) > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3703 > 0x7c8eb6 : lg %r8,360(%r12) > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3709 > 0x7c8ebc : je 0x7c8ec4 > 0x7c8ec0 : j 0x7c8ec2 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3711 > 0x7c8ec4 : lg %r1,568(%r7) > 0x7c8eca : llgf %r1,216(%r1) > /home/maier/kernel/linux-vanilla/./include/linux/slab.h: 495 > 0x7c8ed0 : lgfi %r3,37781696 > 0x7c8ed6 : la %r2,184(%r1) > 0x7c8eda : brasl %r14,0x325e38 <__kmalloc> > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3713 > 0x7c8ee0 : lgr %r11,%r2 > 0x7c8ee4 : cgij %r2,0,8,0x7c9234 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3725 fc_req_to_bsgjob() req->special = job; > 0x7c8eea : stg %r2,248(%r12) > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3726 > 0x7c8ef0 : stg %r6,0(%r2) > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3727 > 0x7c8ef6 : stg %r10,8(%r2) > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3728 fc_req_to_bsgjob() job->req = req; > 0x7c8efc : stg %r12,24(%r2) > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3729 > 0x7c8f02 : lg %r1,568(%r7) > 0x7c8f08 : lt %r1,216(%r1) > 0x7c8f0e : je 0x7c8f1c > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3730 > 0x7c8f12 : la %r1,184(%r2) > 0x7c8f16 : stg %r1,176(%r2) > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3731 > 0x7c8f1c : larl %r4,0x2054808 > 0x7c8f22 : larl %r3,0xbddbd8 > 0x7c8f28 : la %r2,32(%r11) > 0x7c8f2c : brasl %r14,0x1b7ac8 <__raw_spin_lock_init> > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3733 > 0x7c8f32 : llh %r1,288(%r12) > 0x7c8f38 : st %r1,136(%r11) > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3735 > 0x7c8f3c : mvhi 140(%r11),96 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3737 > 0x7c8f42 : ltg %r1,104(%r12) > 0x7c8f48 : jne 0x7c8f56 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3742 > 0x7c8f4c : cgij %r8,0,6,0x7c8f84 > 0x7c8f52 : j 0x7c8f6e > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3738 > 0x7c8f56 : lgr %r3,%r12 > 0x7c8f5a : la %r2,144(%r11) > 0x7c8f5e : brasl %r14,0x7c56c8 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3739 > 0x7c8f64 : cij %r2,0,8,0x7c8f4c > 0x7c8f6a : j 0x7c900e > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3747 > 0x7c8f6e : larl %r1,0x7c5780 > 0x7c8f74 : stg %r1,112(%r11) > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3748 > 0x7c8f7a : cgij %r10,0,6,0x7c8fa6 > 0x7c8f80 : j 0x7c8fd2 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3742 > 0x7c8f84 : ltg %r1,104(%r8) > 0x7c8f8a : je 0x7c8f6e > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3743 > 0x7c8f8e : lgr %r3,%r8 > 0x7c8f92 : la %r2,160(%r11) > 0x7c8f96 : brasl %r14,0x7c56c8 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3744 > 0x7c8f9c : cij %r2,0,8,0x7c8f6e > 0x7c8fa2 : j 0x7c9002 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3749 > 0x7c8fa6 : lg %r2,160(%r15) > 0x7c8fac : stg %r2,16(%r11) > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3752 > 0x7c8fb2 : brasl %r14,0x787968 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3754 > 0x7c8fb8 : mvhi 108(%r11),1 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3979 fc_bsg_request_handler() job = req->special; > 0x7c8fbe : lg %r12,248(%r12) crash> struct -od request.special struct request { [248] void *special; } ******************************************************************** so above %r12 did contain req, below it contains job. since we could deref req further up it must have been non-NULL and pointing to a mapped page, but req->special is NULL here? well, req could even have been NULL and we read from address 248 in low core here which does not trigger a page fault (only on write to low core). crash> x/g 248 0xf8 <_text+248>: 0x0 ******************************************************************** > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3982 > 0x7c8fc4 : l %r1,136(%r12) > 0x7c8fc8 : clij %r1,3,12,0x7c901c > 0x7c8fce : j 0x7c905c > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3751 > 0x7c8fd2 : lg %r1,168(%r15) > 0x7c8fd8 : stg %r1,16(%r11) > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3752 > 0x7c8fde : lgr %r2,%r1 > 0x7c8fe2 : brasl %r14,0x787968 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3754 > 0x7c8fe8 : mvhi 108(%r11),1 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3979 > 0x7c8fee : lg %r12,248(%r12) > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3982 > 0x7c8ff4 : l %r1,136(%r12) > 0x7c8ff8 : clij %r1,3,12,0x7c901c > 0x7c8ffe : j 0x7c90f4 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3760 > 0x7c9002 : lg %r2,152(%r11) > 0x7c9008 : brasl %r14,0x328ff0 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3762 > 0x7c900e : lgr %r2,%r11 > 0x7c9012 : brasl %r14,0x328ff0 > 0x7c9018 : j 0x7c9234 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3983 > 0x7c901c : clfhsi 140(%r12),3 > 0x7c9022 : jh 0x7c902a > 0x7c9026 : j 0x7c9028 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3984 > 0x7c902a : lg %r1,128(%r12) > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3985 > 0x7c9030 : mvhi 4(%r1),0 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3986 > 0x7c9036 : mvhi 0(%r1),-42 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3988 > 0x7c903c : lgr %r2,%r12 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3987 > 0x7c9040 : mvhi 140(%r12),4 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3988 > 0x7c9046 : brasl %r14,0x7c5780 > /home/maier/kernel/linux-vanilla/./include/linux/spinlock.h: 332 > 0x7c904c : lg %r2,2600(%r9) > 0x7c9052 : brasl %r14,0x9a4280 <_raw_spin_lock_irq> > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3990 > 0x7c9058 : j 0x7c8e48 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3892 > 0x7c905c : lg %r2,120(%r12) > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3887 > 0x7c9062 : lg %r11,128(%r12) > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3892 > 0x7c9068 : l %r2,0(%r2) > 0x7c906c : iilf %r3,1073741825 > 0x7c9072 : crj %r2,%r3,8,0x7c9088 > 0x7c9078 : iilf %r3,1073741826 > 0x7c907e : crj %r2,%r3,8,0x7c9090 > 0x7c9084 : j 0x7c90d2 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3894 > 0x7c9088 : lhi %r2,5 > 0x7c908c : j 0x7c9094 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3898 > 0x7c9090 : lhi %r2,16 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3901 > 0x7c9094 : lt %r3,144(%r12) > 0x7c909a : je 0x7c90da > 0x7c909e : lt %r3,160(%r12) > 0x7c90a4 : je 0x7c90da > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3913 > 0x7c90a8 : clrj %r2,%r1,2,0x7c90e2 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3918 > 0x7c90ae : lg %r1,512(%r6) > 0x7c90b4 : lg %r1,568(%r1) > 0x7c90ba : lg %r1,192(%r1) > 0x7c90c0 : lgr %r2,%r12 > 0x7c90c4 : basr %r14,%r1 > 0x7c90c6 : lr %r1,%r2 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3919 > 0x7c90c8 : cij %r2,0,6,0x7c90e6 > 0x7c90ce : j 0x7c9248 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3908 > 0x7c90d2 : lhi %r1,-53 > 0x7c90d6 : j 0x7c90e6 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3903 > 0x7c90da : lhi %r1,-22 > 0x7c90de : j 0x7c90e6 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3914 > 0x7c90e2 : lhi %r1,-42 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3924 > 0x7c90e6 : clfhsi 140(%r12),3 > 0x7c90ec : jh 0x7c91ec > 0x7c90f0 : j 0x7c90f2 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3785 fc_bsg_host_dispatch() struct fc_bsg_request *bsg_request = job->request; > 0x7c90f4 : lg %r3,120(%r12) > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3786 fc_bsg_host_dispatch() struct fc_bsg_reply *bsg_reply = job->reply; > 0x7c90fa : lg %r11,128(%r12) load content of address in %r12 with displacement 128 into %r11. so presumably job->reply is NULL. due to funny inlining incl. fc_bsg_host_dispatch(), it's tricky to backtrack where job in %r12 came from and what happened to it on the way. %r11 is not clobbered until used below where the page fault happens. displacement is consistent: crash> struct -od fc_bsg_job struct fc_bsg_job { [0] struct Scsi_Host *shost; [8] struct fc_rport *rport; [16] struct device *dev; [24] struct request *req; [32] spinlock_t job_lock; [104] unsigned int state_flags; [108] unsigned int ref_cnt; [112] void (*job_done)(struct fc_bsg_job *); [120] struct fc_bsg_request *request; [128] struct fc_bsg_reply *reply; [136] unsigned int request_len; [140] unsigned int reply_len; [144] struct bsg_buffer request_payload; [160] struct bsg_buffer reply_payload; [176] void *dd_data; } SIZE: 184 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3791 > 0x7c9100 : l %r2,0(%r3) > 0x7c9104 : clfi %r2,2147483651 > 0x7c910a : je 0x7c913e > 0x7c910e : jh 0x7c9122 > 0x7c9112 : iilf %r3,2147483649 > 0x7c9118 : clrj %r2,%r3,10,0x7c9194 > 0x7c911e : j 0x7c91c2 > 0x7c9122 : iilf %r4,2147483652 > 0x7c9128 : crj %r2,%r4,8,0x7c9156 > 0x7c912e : iilf %r4,2147483903 > 0x7c9134 : crj %r2,%r4,8,0x7c9172 > 0x7c913a : j 0x7c91c2 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3803 > 0x7c913e : lt %r2,144(%r12) > 0x7c9144 : je 0x7c91ca > 0x7c9148 : lt %r2,160(%r12) > 0x7c914e : je 0x7c91ca > 0x7c9152 : j 0x7c9194 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3813 > 0x7c9156 : lt %r2,144(%r12) > 0x7c915c : je 0x7c91ca > 0x7c9160 : lt %r2,160(%r12) > 0x7c9166 : je 0x7c91ca > 0x7c916a : lhi %r2,20 > 0x7c916e : j 0x7c9198 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3822 > 0x7c9172 : lg %r2,504(%r6) > 0x7c9178 : ltg %r2,304(%r2) > 0x7c917e : je 0x7c91d2 > 0x7c9182 : cg %r2,4(%r3) > 0x7c9188 : jne 0x7c91d2 > 0x7c918c : lhi %r2,12 > 0x7c9190 : j 0x7c9198 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3791 > 0x7c9194 : lhi %r2,8 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3836 > 0x7c9198 : clrj %r2,%r1,2,0x7c91da > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3841 > 0x7c919e : lg %r1,512(%r6) > 0x7c91a4 : lg %r1,568(%r1) > 0x7c91aa : lg %r1,192(%r1) > 0x7c91b0 : lgr %r2,%r12 > 0x7c91b4 : basr %r14,%r1 > 0x7c91b6 : lr %r1,%r2 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3842 > 0x7c91b8 : cij %r2,0,6,0x7c91de > 0x7c91be : j 0x7c9248 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3831 > 0x7c91c2 : lhi %r1,-53 > 0x7c91c6 : j 0x7c91de > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3805 > 0x7c91ca : lhi %r1,-22 > 0x7c91ce : j 0x7c91de > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3825 > 0x7c91d2 : lhi %r1,-3 > 0x7c91d6 : j 0x7c91de > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3837 > 0x7c91da : lhi %r1,-42 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3847 fc_bsg_host_dispatch() fail_host_msg: /* return the errno failure code as the only status */ BUG_ON(job->reply_len < sizeof(uint32_t)); > 0x7c91de : clfhsi 140(%r12),3 > 0x7c91e4 : jh 0x7c91ec > 0x7c91e8 : j 0x7c91ea > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3849 bsg_reply->result = ret; > 0x7c91ec : st %r1,0(%r11) that store causes the kernel page fault because %r11 is NULL and with displacement 0 it still is NULL. > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3848 bsg_reply->reply_payload_rcv_len = 0; > 0x7c91f0 : mvhi 4(%r11),0 if we would have gotten this far: 16-bit signed immediate 0 is extended to 4-bytes and stored to where %r11 with displacement 4 points to. displacements nicely match structure fields: crash> struct -od fc_bsg_reply struct fc_bsg_reply { [0] uint32_t result; [4] uint32_t reply_payload_rcv_len; union { struct fc_bsg_host_vendor_reply vendor_reply; struct fc_bsg_ctels_reply ctels_reply; [8] } reply_data; } SIZE: 16 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3850 job->reply_len = sizeof(uint32_t); > 0x7c91f6 : mvhi 140(%r12),4 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3851 > 0x7c91fc : lgr %r2,%r12 > 0x7c9200 : brasl %r14,0x7c5780 source code is based on > $ git log --graph --oneline > * 271c1723d9c8 scsi: don't use fc_bsg_job::request and fc_bsg_job::reply directly > * a3c95a6c69e4 scsi: Get rid of struct fc_bsg_buffer > * 1573d2caf713 Merge branch 'parisc-4.9-2' of git://git.kernel.org/pub/scm/linux/kernel/git/deller/parisc-linux -- Mit freundlichen Grüßen / Kind regards Steffen Maier Linux on z Systems Development IBM Deutschland Research & Development GmbH Vorsitzende des Aufsichtsrats: Martina Koederitz Geschaeftsfuehrung: Dirk Wittkopp Sitz der Gesellschaft: Boeblingen Registergericht: Amtsgericht Stuttgart, HRB 243294 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3t54wn0SyqzDvTN for ; Sat, 29 Oct 2016 00:53:44 +1100 (AEDT) Received: from pps.filterd (m0098421.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.17/8.16.0.17) with SMTP id u9SDnJH1104767 for ; Fri, 28 Oct 2016 09:53:42 -0400 Received: from e06smtp07.uk.ibm.com (e06smtp07.uk.ibm.com [195.75.94.103]) by mx0a-001b2d01.pphosted.com with ESMTP id 26c35fyutt-1 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT) for ; Fri, 28 Oct 2016 09:53:41 -0400 Received: from localhost by e06smtp07.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Fri, 28 Oct 2016 14:53:39 +0100 Received: from b06cxnps4074.portsmouth.uk.ibm.com (d06relay11.portsmouth.uk.ibm.com [9.149.109.196]) by d06dlp03.portsmouth.uk.ibm.com (Postfix) with ESMTP id 7172A1B08067 for ; Fri, 28 Oct 2016 14:55:45 +0100 (BST) Received: from d06av01.portsmouth.uk.ibm.com (d06av01.portsmouth.uk.ibm.com [9.149.37.212]) by b06cxnps4074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id u9SDrbkq30605452 for ; Fri, 28 Oct 2016 13:53:37 GMT Received: from d06av01.portsmouth.uk.ibm.com (localhost [127.0.0.1]) by d06av01.portsmouth.uk.ibm.com (8.14.4/8.14.4/NCO v10.0 AVout) with ESMTP id u9SDrZKE015055 for ; Fri, 28 Oct 2016 07:53:37 -0600 Subject: Re: [PATCH v2 02/16] scsi: don't use fc_bsg_job::request and fc_bsg_job::reply directly To: Hannes Reinecke , Johannes Thumshirn , Ulrich Weigand , Andreas Krebbel References: <2ea07f3f-88eb-b795-fa37-a223bf80e581@linux.vnet.ibm.com> <20161013162405.aoxy3bdkc4bqtwsk@linux-x5ow.site> <4b411836-e76f-b67a-3d49-ad3d51b8f216@linux.vnet.ibm.com> Cc: "Martin K . Petersen" , Christoph Hellwig , Linux Kernel Mailinglist , Linux SCSI Mailinglist , Martin Schwidefsky , Heiko Carstens , Anil Gurumurthy , Sudarsana Kalluru , "James E.J. Bottomley" , Tyrel Datwyler , Benjamin Herrenschmidt , Paul Mackerras , Michael Ellerman , Johannes Thumshirn , James Smart , Dick Kennedy , "supporter:QLOGIC QLA2XXX FC-SCSI DRIVER" , "open list:S390 ZFCP DRIVER" , "open list:LINUX FOR POWERPC (32-BIT AND 64-BIT)" , "open list:FCOE SUBSYSTEM (libfc, libfcoe, fcoe)" , Richard Biener From: Steffen Maier Date: Fri, 28 Oct 2016 15:53:33 +0200 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Message-Id: <132d8dc4-e8d0-6eba-9ae2-4a7e2c9a589b@linux.vnet.ibm.com> List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , On 10/28/2016 01:31 PM, Hannes Reinecke wrote: > On 10/28/2016 11:53 AM, Steffen Maier wrote: >> On 10/13/2016 06:24 PM, Johannes Thumshirn wrote: >>> On Thu, Oct 13, 2016 at 05:15:25PM +0200, Steffen Maier wrote: >>>> I'm puzzled. >>>> >>>> $ git bisect start fc_bsg master >> >>>>> 3087864ce3d7282f59021245d8a5f83ef1caef18 is the first bad commit >>>>> commit 3087864ce3d7282f59021245d8a5f83ef1caef18 >>>>> Author: Johannes Thumshirn >>>>> Date: Wed Oct 12 15:06:28 2016 +0200 >>>>> >>>>> scsi: don't use fc_bsg_job::request and fc_bsg_job::reply direc= tly >>>>> >>>>> Don't use fc_bsg_job::request and fc_bsg_job::reply directly, >>>>> but use >>>>> helper variables bsg_request and bsg_reply. This will be >>>>> helpfull when >>>>> transitioning to bsg-lib. >>>>> >>>>> Signed-off-by: Johannes Thumshirn >>>>> >>>>> :040000 040000 140c4b6829d5cfaec4079716e0795f63f8bc3bd2 >>>>> 0d9fe225615679550be91fbd9f84c09ab1e280fc M drivers >>>> >>>> From there (on the reverse bisect path) I get the following Oops, >>>> except for the full patch set having another stack trace as in my >>>> previous >>>> mail (dying in zfcp code). >>> >>> [...] >>> >>>>> @@ -3937,6 +3944,7 @@ fc_bsg_request_handler(struct request_queue >>>>> *q, struct Scsi_Host *shost, >>>>> struct request *req; >>>>> struct fc_bsg_job *job; >>>>> enum fc_dispatch_result ret; >>>>> + struct fc_bsg_reply *bsg_reply; >>>>> >>>>> if (!get_device(dev)) >>>>> return; >>>>> @@ -3973,8 +3981,9 @@ fc_bsg_request_handler(struct request_queue >>>>> *q, struct Scsi_Host *shost, >>>>> /* check if we have the msgcode value at least */ >>>>> if (job->request_len < sizeof(uint32_t)) { >>>>> BUG_ON(job->reply_len < sizeof(uint32_t)); >>>>> - job->reply->reply_payload_rcv_len =3D 0; >>>>> - job->reply->result =3D -ENOMSG; >>>>> + bsg_reply =3D job->reply; >>>>> + bsg_reply->reply_payload_rcv_len =3D 0; >>>>> + bsg_reply->result =3D -ENOMSG; >> >> Compiler optimization re-ordered above two lines and the first pointer= >> derefence is bsg_reply->result [field offset 0] where bsg_reply is NUL= L. >> The assignment tries to write to memory at address NULL causing the >> kernel page fault. >> > I spoke to our compiler people, and they strongly believed this not to > be the case. Or, put it the other way round, if such a thing would > happen it would be a compiler issue. > > Have you checked the compiler output? I just mentioned the compiler optimization to explain why the assembler=20 code visible in the panic dies at bsg_reply->result =3D -ENOMSG and not a= t=20 bsg_reply->reply_payload_rcv_len =3D 0. I don't think it makes a=20 difference regarding the issue, which remains a NULL pointer dereference = with bsg_reply either way, which I doubt is caused by compiler output.=20 But then again, see further down below. > [ 46.942560] Krnl PSW : 0704e00180000000 00000000007c91ec[ 46.94257= 4] (fc_bsg_request_handler+0x404/0x4b0) > [ 46.942579] > [ 46.942583] R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:2= PM:000: > [ 46.942598] RI:0 EA:3 > [ 46.942601] > [ 46.942601] Krnl GPRS: 0000000000000000 00000000ffffffcb 00000000000= 00000 0000000080000001 > [ 46.942603] 00000000007c8fe8 0000000064398c68 0000000069f= 967e8 000000006a3d8008 > [ 46.942605] 000000006a5e02c8 00000000698b5490 00000000000= 00000 0000000000000000 =20 %r11 is NULL > [ 46.942607] 000000006a9ef5f8 0000000000a36840 00000000007= c8fe8 000000005d2efa00 > [ 46.942619] Krnl Code: 00000000007c91de: e55dc08c0003 clfhsi = 140(%r12),3[ 46.942622] > [ 46.942622] 00000000007c91e4: a7240004 brc = 2,7c91ec > #00000000007c91e8: a7f40001 brc = 15,7c91ea[ 46.942629] > [ 46.942629] >00000000007c91ec: 5010b000 st = %r1,0(%r11) > 00000000007c91f0: e54cb0040000 mvhi = 4(%r11),0[ 46.942635] > [ 46.942635] 00000000007c91f6: e54cc08c0004 mvhi = 140(%r12),4 > 00000000007c91fc: b904002c lgr = %r2,%r12[ 46.942643] > [ 46.942643] 00000000007c9200: c0e5ffffe2c0 brasl = %r14,7c5780 > [ 46.942646] > [ 46.942647] Call Trace: > [ 46.942650] ([<00000000007c8fe8>] fc_bsg_request_handler+0x200/0x4b0= ) > [ 46.942656] ([<00000000006b8e0a>] __blk_run_queue+0x52/0x68) > [ 46.942661] ([<00000000006c549a>] blk_execute_rq_nowait+0xf2/0x110) > [ 46.942664] ([<00000000006c557a>] blk_execute_rq+0xa2/0x110) > [ 46.942668] ([<00000000006de0ee>] bsg_ioctl+0x1f6/0x268) > [ 46.942675] ([<000000000036ca20>] do_vfs_ioctl+0x680/0x6d8) > [ 46.942677] ([<000000000036caf4>] SyS_ioctl+0x7c/0xb0) > [ 46.942685] ([<00000000009a541e>] system_call+0xd6/0x270) > [ 46.942687] INFO: lockdep is turned off. > [ 46.942688] Last Breaking-Event-Address: > [ 46.942692] [<00000000007c91e4>] fc_bsg_request_handler+0x3fc/0x4b0= > [ 46.942696] [ 46.942698] Kernel panic - not syncing: Fatal except= ion: panic_on_oops all the following was written from bottom to top: > crash> dis -l fc_bsg_request_handler > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3943= static void fc_bsg_request_handler(struct request_queue *q, struct Scsi_Host *shost, struct fc_rport *rport, struct device *dev) { > 0x7c8de8 : brcl 0,0x7c8de8 > 0x7c8dee : stmg %r6,%r15,72(%r15) > 0x7c8df4 : larl %r13,0xa36840 > 0x7c8dfa : tmll %r15,16256 > 0x7c8dfe : lgr %r14,%r15 > 0x7c8e02 : je 0x7c8e04 > 0x7c8e06 : lay %r15,-112(%r15) > 0x7c8e0c : stg %r14,152(%r15) > 0x7c8e12 : lgr %r9,%r2 > 0x7c8e16 : stg %r5,176(%r15) > 0x7c8e1c : lgr %r2,%r5 > 0x7c8e20 : lgr %r6,%r3 > 0x7c8e24 : lgr %r10,%r4 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3949= > 0x7c8e28 : brasl %r14,0x787968 > 0x7c8e2e : cgij %r2,0,8,0x7c9288 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3751= there is some confusing inlining of part of fc_req_to_bsgjob > 0x7c8e34 : la %r1,960(%r6) > 0x7c8e38 : stg %r1,168(%r15) > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3749= > 0x7c8e3e : la %r1,96(%r10) > 0x7c8e42 : stg %r1,160(%r15) > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3953= > 0x7c8e48 : cgij %r10,0,8,0x7c9270 > 0x7c8e4e : clc 4(4,%r13),40(%r10) > 0x7c8e54 : jne 0x7c9258 > 0x7c8e58 : tm 72(%r10),4 > 0x7c8e5c : jne 0x7c9258 > 0x7c8e60 : j 0x7c920a > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3961= > 0x7c8e64 : clc 0(4,%r13),40(%r10) > 0x7c8e6a : je 0x7c8e9e > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3962= fc_bsg_request_handler() req->errors =3D -ENXIO; > 0x7c8e6e : mvhi 260(%r12),-6 crash> struct -od request.errors struct request { [260] int errors; } ******************************************************************** BUT this seems the first time %r12 is used in fc_bsg_request_handler(), especially I seem to miss %r12 being initalized with anything. But then again I'm not at all well versed in disassembly. Maybe fc_bsg_request_handler() is itself in turn inlined and I would=20 need to start disassembling even earlier to get to %r12 init? s390x ELF ABI says %r12: usage: Local variable, commonly used as GOT pointer; call effect: saved. Even if it wasn't initialized and remained NULL below why did it not=20 already page fault at above instruction? Silly me, we did not execute=20 this instruction as it's "if" conditional. This makes me wonder even=20 more where the content of %r12 comes from. Ulli, Andreas, could you please shed some light on this? ******************************************************************** > /home/maier/kernel/linux-vanilla/./include/linux/spinlock.h: 357 > 0x7c8e74 : lg %r2,2600(%r9) > 0x7c8e7a : brasl %r14,0x9a46d0 <_raw_spi= n_unlock_irq> > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3964= > 0x7c8e80 : lgr %r2,%r12 > 0x7c8e84 : lghi %r3,-6 > 0x7c8e88 : brasl %r14,0x6be2f0 > /home/maier/kernel/linux-vanilla/./include/linux/spinlock.h: 332 > 0x7c8e8e : lg %r2,2600(%r9) > 0x7c8e94 : brasl %r14,0x9a4280 <_raw_spi= n_lock_irq> > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3966= > 0x7c8e9a : j 0x7c8e48 > /home/maier/kernel/linux-vanilla/./include/linux/spinlock.h: 357 > 0x7c8e9e : lg %r2,2600(%r9) > 0x7c8ea4 : brasl %r14,0x9a46d0 <_raw_spi= n_unlock_irq> > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3709= > 0x7c8eaa : ltg %r1,248(%r12) > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3702= > 0x7c8eb0 : lg %r7,512(%r6) > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3703= > 0x7c8eb6 : lg %r8,360(%r12) > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3709= > 0x7c8ebc : je 0x7c8ec4 > 0x7c8ec0 : j 0x7c8ec2 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3711= > 0x7c8ec4 : lg %r1,568(%r7) > 0x7c8eca : llgf %r1,216(%r1) > /home/maier/kernel/linux-vanilla/./include/linux/slab.h: 495 > 0x7c8ed0 : lgfi %r3,37781696 > 0x7c8ed6 : la %r2,184(%r1) > 0x7c8eda : brasl %r14,0x325e38 <__kmallo= c> > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3713= > 0x7c8ee0 : lgr %r11,%r2 > 0x7c8ee4 : cgij %r2,0,8,0x7c9234 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3725= fc_req_to_bsgjob() req->special =3D job; > 0x7c8eea : stg %r2,248(%r12) > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3726= > 0x7c8ef0 : stg %r6,0(%r2) > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3727= > 0x7c8ef6 : stg %r10,8(%r2) > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3728= fc_req_to_bsgjob() job->req =3D req; > 0x7c8efc : stg %r12,24(%r2) > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3729= > 0x7c8f02 : lg %r1,568(%r7) > 0x7c8f08 : lt %r1,216(%r1) > 0x7c8f0e : je 0x7c8f1c > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3730= > 0x7c8f12 : la %r1,184(%r2) > 0x7c8f16 : stg %r1,176(%r2) > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3731= > 0x7c8f1c : larl %r4,0x2054808 <= proc_scsi+0x48> > 0x7c8f22 : larl %r3,0xbddbd8 > 0x7c8f28 : la %r2,32(%r11) > 0x7c8f2c : brasl %r14,0x1b7ac8 <= __raw_spin_lock_init> > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3733= > 0x7c8f32 : llh %r1,288(%r12) > 0x7c8f38 : st %r1,136(%r11) > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3735= > 0x7c8f3c : mvhi 140(%r11),96 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3737= > 0x7c8f42 : ltg %r1,104(%r12) > 0x7c8f48 : jne 0x7c8f56 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3742= > 0x7c8f4c : cgij %r8,0,6,0x7c8f8= 4 > 0x7c8f52 : j 0x7c8f6e > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3738= > 0x7c8f56 : lgr %r3,%r12 > 0x7c8f5a : la %r2,144(%r11) > 0x7c8f5e : brasl %r14,0x7c56c8 <= fc_bsg_map_buffer> > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3739= > 0x7c8f64 : cij %r2,0,8,0x7c8f4= c > 0x7c8f6a : j 0x7c900e > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3747= > 0x7c8f6e : larl %r1,0x7c5780 > 0x7c8f74 : stg %r1,112(%r11) > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3748= > 0x7c8f7a : cgij %r10,0,6,0x7c8f= a6 > 0x7c8f80 : j 0x7c8fd2 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3742= > 0x7c8f84 : ltg %r1,104(%r8) > 0x7c8f8a : je 0x7c8f6e > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3743= > 0x7c8f8e : lgr %r3,%r8 > 0x7c8f92 : la %r2,160(%r11) > 0x7c8f96 : brasl %r14,0x7c56c8 <= fc_bsg_map_buffer> > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3744= > 0x7c8f9c : cij %r2,0,8,0x7c8f6= e > 0x7c8fa2 : j 0x7c9002 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3749= > 0x7c8fa6 : lg %r2,160(%r15) > 0x7c8fac : stg %r2,16(%r11) > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3752= > 0x7c8fb2 : brasl %r14,0x787968 <= get_device> > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3754= > 0x7c8fb8 : mvhi 108(%r11),1 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3979= fc_bsg_request_handler() job =3D req->special; > 0x7c8fbe : lg %r12,248(%r12) crash> struct -od request.special struct request { [248] void *special; } ******************************************************************** so above %r12 did contain req, below it contains job. since we could deref req further up it must have been non-NULL and=20 pointing to a mapped page, but req->special is NULL here? well, req could even have been NULL and we read from address 248 in low=20 core here which does not trigger a page fault (only on write to low core)= =2E crash> x/g 248 0xf8 <_text+248>: 0x0 ******************************************************************** > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3982= > 0x7c8fc4 : l %r1,136(%r12) > 0x7c8fc8 : clij %r1,3,12,0x7c90= 1c > 0x7c8fce : j 0x7c905c > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3751= > 0x7c8fd2 : lg %r1,168(%r15) > 0x7c8fd8 : stg %r1,16(%r11) > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3752= > 0x7c8fde : lgr %r2,%r1 > 0x7c8fe2 : brasl %r14,0x787968 <= get_device> > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3754= > 0x7c8fe8 : mvhi 108(%r11),1 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3979= > 0x7c8fee : lg %r12,248(%r12) > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3982= > 0x7c8ff4 : l %r1,136(%r12) > 0x7c8ff8 : clij %r1,3,12,0x7c90= 1c > 0x7c8ffe : j 0x7c90f4 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3760= > 0x7c9002 : lg %r2,152(%r11) > 0x7c9008 : brasl %r14,0x328ff0 <= kfree> > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3762= > 0x7c900e : lgr %r2,%r11 > 0x7c9012 : brasl %r14,0x328ff0 <= kfree> > 0x7c9018 : j 0x7c9234 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3983= > 0x7c901c : clfhsi 140(%r12),3 > 0x7c9022 : jh 0x7c902a > 0x7c9026 : j 0x7c9028 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3984= > 0x7c902a : lg %r1,128(%r12) > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3985= > 0x7c9030 : mvhi 4(%r1),0 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3986= > 0x7c9036 : mvhi 0(%r1),-42 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3988= > 0x7c903c : lgr %r2,%r12 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3987= > 0x7c9040 : mvhi 140(%r12),4 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3988= > 0x7c9046 : brasl %r14,0x7c5780 <= fc_bsg_jobdone> > /home/maier/kernel/linux-vanilla/./include/linux/spinlock.h: 332 > 0x7c904c : lg %r2,2600(%r9) > 0x7c9052 : brasl %r14,0x9a4280 <= _raw_spin_lock_irq> > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3990= > 0x7c9058 : j 0x7c8e48 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3892= > 0x7c905c : lg %r2,120(%r12) > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3887= > 0x7c9062 : lg %r11,128(%r12) > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3892= > 0x7c9068 : l %r2,0(%r2) > 0x7c906c : iilf %r3,1073741825 > 0x7c9072 : crj %r2,%r3,8,0x7c9= 088 > 0x7c9078 : iilf %r3,1073741826 > 0x7c907e : crj %r2,%r3,8,0x7c9= 090 > 0x7c9084 : j 0x7c90d2 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3894= > 0x7c9088 : lhi %r2,5 > 0x7c908c : j 0x7c9094 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3898= > 0x7c9090 : lhi %r2,16 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3901= > 0x7c9094 : lt %r3,144(%r12) > 0x7c909a : je 0x7c90da > 0x7c909e : lt %r3,160(%r12) > 0x7c90a4 : je 0x7c90da > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3913= > 0x7c90a8 : clrj %r2,%r1,2,0x7c9= 0e2 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3918= > 0x7c90ae : lg %r1,512(%r6) > 0x7c90b4 : lg %r1,568(%r1) > 0x7c90ba : lg %r1,192(%r1) > 0x7c90c0 : lgr %r2,%r12 > 0x7c90c4 : basr %r14,%r1 > 0x7c90c6 : lr %r1,%r2 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3919= > 0x7c90c8 : cij %r2,0,6,0x7c90e= 6 > 0x7c90ce : j 0x7c9248 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3908= > 0x7c90d2 : lhi %r1,-53 > 0x7c90d6 : j 0x7c90e6 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3903= > 0x7c90da : lhi %r1,-22 > 0x7c90de : j 0x7c90e6 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3914= > 0x7c90e2 : lhi %r1,-42 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3924= > 0x7c90e6 : clfhsi 140(%r12),3 > 0x7c90ec : jh 0x7c91ec > 0x7c90f0 : j 0x7c90f2 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3785= fc_bsg_host_dispatch() struct fc_bsg_request *bsg_request =3D job->request; > 0x7c90f4 : lg %r3,120(%r12) > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3786= fc_bsg_host_dispatch() struct fc_bsg_reply *bsg_reply =3D job->reply; > 0x7c90fa : lg %r11,128(%r12) load content of address in %r12 with displacement 128 into %r11. so presumably job->reply is NULL. due to funny inlining incl. fc_bsg_host_dispatch(), it's tricky to=20 backtrack where job in %r12 came from and what happened to it on the way.= %r11 is not clobbered until used below where the page fault happens. displacement is consistent: crash> struct -od fc_bsg_job struct fc_bsg_job { [0] struct Scsi_Host *shost; [8] struct fc_rport *rport; [16] struct device *dev; [24] struct request *req; [32] spinlock_t job_lock; [104] unsigned int state_flags; [108] unsigned int ref_cnt; [112] void (*job_done)(struct fc_bsg_job *); [120] struct fc_bsg_request *request; [128] struct fc_bsg_reply *reply; [136] unsigned int request_len; [140] unsigned int reply_len; [144] struct bsg_buffer request_payload; [160] struct bsg_buffer reply_payload; [176] void *dd_data; } SIZE: 184 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3791= > 0x7c9100 : l %r2,0(%r3) > 0x7c9104 : clfi %r2,2147483651 > 0x7c910a : je 0x7c913e > 0x7c910e : jh 0x7c9122 > 0x7c9112 : iilf %r3,2147483649 > 0x7c9118 : clrj %r2,%r3,10,0x7c= 9194 > 0x7c911e : j 0x7c91c2 > 0x7c9122 : iilf %r4,2147483652 > 0x7c9128 : crj %r2,%r4,8,0x7c9= 156 > 0x7c912e : iilf %r4,2147483903 > 0x7c9134 : crj %r2,%r4,8,0x7c9= 172 > 0x7c913a : j 0x7c91c2 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3803= > 0x7c913e : lt %r2,144(%r12) > 0x7c9144 : je 0x7c91ca > 0x7c9148 : lt %r2,160(%r12) > 0x7c914e : je 0x7c91ca > 0x7c9152 : j 0x7c9194 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3813= > 0x7c9156 : lt %r2,144(%r12) > 0x7c915c : je 0x7c91ca > 0x7c9160 : lt %r2,160(%r12) > 0x7c9166 : je 0x7c91ca > 0x7c916a : lhi %r2,20 > 0x7c916e : j 0x7c9198 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3822= > 0x7c9172 : lg %r2,504(%r6) > 0x7c9178 : ltg %r2,304(%r2) > 0x7c917e : je 0x7c91d2 > 0x7c9182 : cg %r2,4(%r3) > 0x7c9188 : jne 0x7c91d2 > 0x7c918c : lhi %r2,12 > 0x7c9190 : j 0x7c9198 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3791= > 0x7c9194 : lhi %r2,8 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3836= > 0x7c9198 : clrj %r2,%r1,2,0x7c9= 1da > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3841= > 0x7c919e : lg %r1,512(%r6) > 0x7c91a4 : lg %r1,568(%r1) > 0x7c91aa : lg %r1,192(%r1) > 0x7c91b0 : lgr %r2,%r12 > 0x7c91b4 : basr %r14,%r1 > 0x7c91b6 : lr %r1,%r2 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3842= > 0x7c91b8 : cij %r2,0,6,0x7c91d= e > 0x7c91be : j 0x7c9248 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3831= > 0x7c91c2 : lhi %r1,-53 > 0x7c91c6 : j 0x7c91de > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3805= > 0x7c91ca : lhi %r1,-22 > 0x7c91ce : j 0x7c91de > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3825= > 0x7c91d2 : lhi %r1,-3 > 0x7c91d6 : j 0x7c91de > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3837= > 0x7c91da : lhi %r1,-42 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3847= fc_bsg_host_dispatch() fail_host_msg: /* return the errno failure code as the only status */ BUG_ON(job->reply_len < sizeof(uint32_t)); > 0x7c91de : clfhsi 140(%r12),3 > 0x7c91e4 : jh 0x7c91ec > 0x7c91e8 : j 0x7c91ea > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3849= bsg_reply->result =3D ret; > 0x7c91ec : st %r1,0(%r11) that store causes the kernel page fault because %r11 is NULL and with=20 displacement 0 it still is NULL. > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3848= bsg_reply->reply_payload_rcv_len =3D 0; > 0x7c91f0 : mvhi 4(%r11),0 if we would have gotten this far: 16-bit signed immediate 0 is extended to 4-bytes and stored to where=20 %r11 with displacement 4 points to. displacements nicely match structure fields: crash> struct -od fc_bsg_reply struct fc_bsg_reply { [0] uint32_t result; [4] uint32_t reply_payload_rcv_len; union { struct fc_bsg_host_vendor_reply vendor_reply; struct fc_bsg_ctels_reply ctels_reply; [8] } reply_data; } SIZE: 16 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3850= job->reply_len =3D sizeof(uint32_t); > 0x7c91f6 : mvhi 140(%r12),4 > /home/maier/kernel/linux-vanilla/drivers/scsi/scsi_transport_fc.c: 3851= > 0x7c91fc : lgr %r2,%r12 > 0x7c9200 : brasl %r14,0x7c5780 <= fc_bsg_jobdone> source code is based on > $ git log --graph --oneline > * 271c1723d9c8 scsi: don't use fc_bsg_job::request and fc_bsg_job::repl= y directly > * a3c95a6c69e4 scsi: Get rid of struct fc_bsg_buffer > * 1573d2caf713 Merge branch 'parisc-4.9-2' of git://git.kernel.org/pu= b/scm/linux/kernel/git/deller/parisc-linux --=20 Mit freundlichen Gr=FC=DFen / Kind regards Steffen Maier Linux on z Systems Development IBM Deutschland Research & Development GmbH Vorsitzende des Aufsichtsrats: Martina Koederitz Geschaeftsfuehrung: Dirk Wittkopp Sitz der Gesellschaft: Boeblingen Registergericht: Amtsgericht Stuttgart, HRB 243294